Prose Regression Testing Across Recursive Draft Passes

Recursive AI writing pipelines need a generate-critique-revise loop governed by prose regression tests. Each revision must be compared with a preserved baseline through deterministic checks for protected material and semantic checks for meaning, framing, and voice. The central design principle is that revision proposes a change, while an independent quality gate decides whether that change is acceptable.

What prose regression testing is and what problem it solves for recursive AI drafting

The review step can introduce the defect. A model asked to improve technical prose can change a command prefix, soften a requirement, or remove a qualification while making the sentences easier to read. These are semantic regressions because the edited text no longer preserves what the source said or required. A style linter can accept that revision because its checks concern surface mechanics.

Prose regression testing applies the software-QA pattern of baseline capture, diff-based comparison, and pass/fail gates to this gap. The baseline preserves the content against which an edit must be judged. The comparison identifies changes, while the gates distinguish permitted revisions from losses of meaning or protected technical detail. This is a preservation test. Its purpose is to let an editor improve expression without granting unrestricted control over the content being expressed.

Agentflow 0.0.4 uses a similar testing approach in a cold-context authoring benchmark with deterministic mutation tests and a model-free CI gate. It does not document a second model changing a command, removing a link, or weakening a qualification. That distinction limits the evidence appropriately. The benchmark supports the use of executable checks around authoring, rather than serving as an observed example of every regression such checks might address.

Why drift compounds across recursive passes rather than cancelling out

When the previous model’s output becomes the next model’s input, its assumptions can become settled premises. Han-yu Wang’s 2026 paper on early epistemic settlement in AI-assisted writing describes how accepted generated representations shape subsequent deliberation. Participants retained suggested framings as they elaborated and reworded ideas. Rephrasing did not necessarily reopen the underlying decision.

Chen et al.’s 2026 ACL paper on multi-turn report revision identifies a related preservation failure. Deep research agents disrupted content that the feedback had not asked them to change and failed to keep earlier edits. Those findings undermine the assumption that revision is a series of local improvements whose benefits simply accumulate. An edit can repair one passage while undoing an accepted change elsewhere. The next pass then inherits both results.

A draft that passes its checks at pass two may have drifted from the original by pass five without any individual transition triggering those checks. Whether cumulative drift has crossed an acceptable threshold matters even when the latest change looks reasonable. Comparison with a preserved baseline keeps that question answerable throughout the loop.

How multi-model and adversarial architectures intensify the regression problem before they solve it

Feng et al.’s position paper, arxiv 2502.04506, argues that one LLM underrepresents real-world data distributions, heterogeneous skills, and pluralistic populations in complex, contextualized, subjective settings. Further training of that model cannot trivially close those gaps. Different models can bring views that a single model does not cover well enough. Diversity has a sound architectural purpose.

But each added model also brings another chance for drift. Each has a stylistic fingerprint, and cascaded revisions can compound framing artifacts that no individual model would produce alone. The number of agents does not establish the quality of their coordination. Without a shared preservation constraint, collaborators can disagree through successive rewrites rather than through explicit judgments about an unchanged source. The result may sound clear and connected while moving farther from the intended argument.

The EACL 2026 DITTO paper adds a separate warning about trusting provenance signals. It shows that a malicious model can generate text containing an authentic-looking watermark of a trusted victim model without access to its secret key. Attribution alone cannot establish trust in a review chain. A testing layer outside the generative chain compares proposed output with a preserved baseline.

The two-layer architecture that separates deterministic checks from semantic judgment

The first layer checks things that have exact answers. Fenced code blocks, inline code, link destinations, file paths, numbers, and Markdown structure can be compared without asking a model to interpret the argument. Agentflow calls these elements “protected technical material” and fails an edit when a protected value changes or disappears. Such checks belong in code because there is nothing to gain from leaving them open to interpretation. A changed command literal is not a stylistic disagreement.

The second layer asks whether the argument’s arc survives, whether a qualification has been softened, and whether the voice has moved away from the baseline. Matsumotory’s August 2026 practice assigns uniquely determined checks to local functions and meaning, logic, and chapter structure to an LLM reviewer. The layers run in sequence. An edit must pass the literal check even if it has semantic approval.

The swack-tools/vale-ai-plugin PR #6, merged September 28, 2026, makes this separation concrete in a labeled regression corpus. Six paired baseline/skill trials include literal checks that reject changed command prefixes and same-line continuations, with semantic review dimensions validated separately. Agentflow also preserves the source by writing the edited result to a sibling file. The editor must never overwrite the only draft copy. The resulting comparison retains both the proposed revision and the material needed for a diff or rollback.

Turning reviewer feedback into permanent rules rather than repeated judgment calls

A review finding becomes more useful when it survives the review session. Matsumotory’s practice converts each point raised by a reviewer into a rule stored in the YAML test suite. Later drafts are checked against that rule automatically, so no one has to remember and raise the same objection again. A previously discovered failure is now part of the acceptance criteria.

This changes the role of feedback within recursion. Without a durable rule, a correction can disappear during a later revision, leaving the same issue to be rediscovered. With a versioned check, that issue remains visible even as the draft and its editors change. Agentflow gives this approach an ordered workflow through file eligibility classification, deterministic preflight, one editor pass into a sibling file, protected-material comparison, and project validation commands. Each step acts as a gate whose definition can be versioned.

The growing suite does not prove that every accepted draft is easier to read. Its bounded claim is recurrence prevention for the failures it captures, subject to the reliability of the checks themselves. That distinction prevents test coverage from becoming a substitute for editorial judgment. A passing result says the draft satisfied the recorded constraints. It does not establish that the constraints cover every weakness in the prose.

The unresolved problem of reviewer consistency when the judge wavers

The semantic layer depends on a judge whose consistency is not yet established. Matsumotory states that the frequency with which the reviewer returns the same judgment for the same input remains unquantified. If a verdict wavers, the list of failing checks becomes harder to trust. This is not merely an inconvenience in presentation. It affects whether the gate can function as an acceptance decision.

The practice includes a limited mitigation for that uncertainty. When the first judgment is not a pass, the check runs twice more and takes a majority vote over three judgments. Whether this produces stable decisions remains unconfirmed. Repetition can expose variability, but repeated calls to one reviewer do not introduce independent perspectives. The same model still supplies the framing and stylistic preferences behind each verdict.

Adversarial multi-model review becomes a quality-gate requirement here. Independent reviewers can bring disagreements to light before one model’s judgment is taken as final. A split verdict is a quality signal because it identifies an ambiguity in the draft, the criterion, or its application. Feng et al.’s argument supplies the theoretical basis for seeking heterogeneous judgment, though it does not prove reviewer stability. Multiple reviewers make disagreement inspectable. They do not eliminate the need to assess the gate itself.

What faithfulness-gated training adds that test suites alone cannot provide

Runtime tests examine what a model has produced. They do not, by themselves, control which examples enter its future training corpus. A 2026 paper on AI-generated race-strategy briefings moves the faithfulness metric upstream as a “faithfulness gate on the corpus” that determines what the model is permitted to learn from.

Applied to brand content pipelines, the same principle makes approved outputs that passed the full regression suite eligible for the training corpus. Outputs with drifted voice, changed facts, or softened qualifications are excluded from ingestion. The suite therefore has a second role beyond accepting individual drafts. Its accumulated review rules also govern which examples can teach the model. A recursive pipeline should require more than the fact that it produced an output before using that output as training material.

Corpus selection also requires attention to provenance. Kirchenbauer and co-authors’ 2026 paper shows that a model trained on watermarked text can itself produce watermarked output. That finding adds a constraint beyond voice quality and factual fidelity. Passing a prose regression suite does not resolve every property of an ingested document. Faithfulness-gated training uses the output gate to check what the model can learn from, while provenance checks cover another way that training material can shape later output.

How do you build an AI writing system that refines its own output automatically?

In a working setup, prose regression testing is part of the tools developers use. Agentflow records the classification field with the writing task rather than inferring eligibility from a filename, then runs deterministic preflight before any model call. One editor pass writes to a sibling file, then protected-material comparison and project validation commands check the new draft. CI discovers the tests, and the artifacts remain versioned and auditable. A failed gate prevents a proposed improvement from being treated as an accepted revision.

The vale-ai-plugin contribution backs up this approach to software testing with 195 tests locally. Its failing-first regressions cover changed command literals, stale review hashes, malformed JSON, and symlink loops. These cases test the machinery around review as well as the text under review. The gate itself needs regression protection. Otherwise, a sound editorial rule can be undermined by a broken comparison or an invalid review result.

The deployable components are a CLI for preflight and verification, an API for semantic review, YAML-defined suites versioned with their content, and sibling-file diffs that preserve the baseline. Adversarial reviewers assess meaning, while faithfulness-gated corpus ingestion controls which accepted outputs become learning material. Human-readable test authorship can fit this workflow too. The March 2026 markdown-based regression pattern uses plain-English steps that an AI agent translates into executable commands, with parallel runs in isolated sessions and no selectors or XPath.

Humans establish the workflow, protected material, semantic criteria, and gates before the recursive loop runs. The system then enforces those constraints without requiring a person to review every line after every pass. Its boundary remains explicit. Automation can detect the recurrence of captured failures, while a human still judges what to fix and whether the prose is good.