Measuring Prose Quality Beyond Readability Scores

Sentence length and syllable count describe the surface of writing, whose deeper properties include whether its choices serve a purpose, its voice holds together, and its argument reaches a clear conclusion. A rigorous assessment framework must measure those deeper properties, using formulas as limited signals alongside editorial judgment.

What readability formulas actually measure — and what they were built for

Flesch-Kincaid, Gunning Fog, and SMOG estimate reading difficulty from surface features of text. They use sentence length and word length, including syllable count, but do not measure comprehension itself. The 2026 Gutenberg extension examines seven formulas built from the same two or three surface quantities, weighted differently. Because those formulas share their ingredients, their scores largely rise and fall together.

That shallowness was a deliberate design choice. These measures were cheap enough to run across an entire corpus and gave users an estimate they could repeat without having to read the text in context. The cross-dataset analysis notes, however, that readability is an overloaded term and that prior work measures it in different ways. A formula therefore answers a narrower question than the word “readable” usually implies in an editorial brief.

Why the formulas persist even though researchers know their limits

The practical appeal of that narrow answer is considerable. A single number is quick to compute, easy to explain, and simple to put into a style guide. An editor under pressure to get work out has a clear standard to use across many drafts. Its legibility helps explain its use even when its limits are understood.

Riley Grossman and Yi Chen’s zero-shot LLM study searched Scopus for peer-reviewed papers published in 2025 and found 337 papers using readability formulas. It found only 9 using model-based approaches, including 2 supervised BERT models, 2 pretrained language model surprisal approaches, and 5 zero-shot LLM approaches. The numbers show how firmly formula-based assessment has taken hold in applied work. In content marketing, readability checkers and SEO platforms reinforce that position by putting a Flesch score into a dashboard. Once a team requires that number before it will release a draft, an easy way to measure writing becomes a rule.

Where surface metrics actively mislead — the quality-readability gap in real text

Policy becomes a problem when a surface score is treated as evidence of achievement. The 2026 Gutenberg data gives a concrete example in Mark Twain’s A Horse’s Tale, which receives a Flesch–Kincaid grade of 7.2. Twain writes in the voices of a horse and a young girl, so the result reflects the register he chose. The score identifies roughly seventh-grade language, but it cannot judge the craft involved in making that language work. Treating it as a quality rating would confuse the means with the accomplishment.

News writing achieves a high Gunning Fog score yet receives lower quality ratings than literary prose with a much lower Flesch-Kincaid grade. Those measures yield numbers with different meanings, but the comparison still defeats the assumption that greater measured complexity means better writing. Birger Moëll’s 2026 literary quality paper finds that readability scores and quality ratings are decoupled. Complexity does not predict quality.

The cross-dataset analysis also finds that information content and topic do much to shape how people judge what they can understand. A syllable-count formula has no way to use either. As a consequence, optimizing a blog post for a target Flesch score can leave it incoherent, voice-free, and factually hollow. The danger is an incomplete diagnosis that encourages changes to visible features while leaving the real failures untouched.

What human readers actually respond to that formulas cannot see

A framework must therefore start with the features that readers judge. The five dimensions are prose craft, depth, voice coherence, structural coherence, and impact. Craft concerns whether language choices do purposeful work, while depth concerns the substance those choices carry. Voice coherence concerns the consistency and individual character of the writing. Structural coherence concerns how its parts fit together, and impact concerns what the whole achieves.

Moëll’s 2026 paper tests a related multidimensional rubric covering prose, depth, craft, voice, and impact. Its LLM evaluators achieve high accuracy in classifying quality tiers that surface formulas cannot distinguish. The paper’s discussion also broadens readability to include syntactic and discourse-level organization, cohesion, and coherence. These properties concern relationships within the text.

Intentionality connects those relationships to editorial judgment. A fragment, an em-dash, or a very short sentence may be a deliberate craft decision. The choice must work for the passage and its reader. Likewise, two passages with the same sentence lengths and syllable counts can have entirely different voices. A quality framework must distinguish purposeful variation from inconsistency.

How model-based assessment closes the gap that formulas leave open

Model-based assessment offers a way to evaluate those contextual properties against explicit criteria. The cross-dataset analysis evaluates fifteen popular readability metrics alongside six model-based metrics. Four model-based metrics keep their place in the top four for rank correlation with human judgments, but the best traditional metric ranks much lower on average. This is evidence about agreement with readers, rather than convenience for the person running the test.

Grossman and Chen’s study tests zero-shot prompting with ten diverse open-source LLMs across multiple datasets. Its method outperforms prior approaches on the large majority of the datasets tested. The study also introduces LAURAE, a hybrid combining LLM scores with formula scores to capture contextual and shallow features. The hybrid outperforms both components alone across languages, text lengths, and amounts of technical language. That finding supports retaining surface measures as inputs without letting them define the entire assessment.

Moëll’s use of DeepSeek R1 with a multidimensional rubric extends the case from reading difficulty to prose quality. The model achieves strong accuracy in classifying quality tiers, showing that craft and voice can be made operational evaluation criteria. The strongest objection is interpretability, because model-based metrics depend on implicitly learned representations whose workings are less transparent than arithmetic. That trade-off matters, but an easily explained score is not sufficient if it measures the wrong property. Formula-only assessment leaves too much of serious content work unexamined.

Why single-model AI output fails the quality dimensions that matter

The same blind spots become particularly visible in single-model AI writing. Its output can be smooth and acceptable to sentence-and-syllable metrics while remaining weak in voice coherence, structural consistency, and individual character. Surface fluency makes those weaknesses less conspicuous while leaving their consequences intact. A draft may read easily from sentence to sentence without developing a distinctive or coherent whole.

Research described in the brief finds that AI texts cluster into tight stylistic groups, while human prose shows much greater dispersion. The distinction is the absence of the “lexical fingerprints” characteristic of human authors. Homogeneity is therefore not simply a matter of a few overused words. It concerns the limited range of stylistic choices through which the writing expresses its subject.

This does not make model-based evaluation contradictory. Generating a draft in one pass and assessing it against an explicit rubric are different tasks. The problem arises when fluency is accepted as evidence that the deeper dimensions have been satisfied. Readability formulas and single-model output share that weakness at the production level, where surface acceptability can become the operative standard. Voice, coherence, and intentionality remain unmanaged unless the system gives them separate tests.

The architectural response — adversarial multi-model review as a quality gate

In an adversarial multi-model system, models draft, critique, and vote on output until it converges on the required quality dimensions. The premise is that self-review leaves a model exposed to its own systematic failures. A second model with a different training distribution can surface patterns the first overlooks. This follows the logic of adversarial testing in security, where independent challenge is part of the design.

Multi-agent orchestration gives that challenge a place in the work of producing a draft. Specialized models can work at the same time, pass output from one stage to the next, and keep context throughout the task. Different tasks can use different models, including one for polished long-form work and a cheaper one for large amounts of short-form work. Quality review can follow the same cascade pattern. Lightweight checks handle the clear cases, and cases that are harder to judge go to more capable judges.

LangChain expresses the underlying relationship as “Agent = Model + Harness”. The harness governs tool permissions, verification loops, and state persistence across the full task lifecycle. For writing, that distinction moves responsibility from a single generated answer to the process that accepts or rejects it. Structural convergence then means more than agreement between models. It means that the text is revised to meet the editorial requirements that have been set.

Voice fidelity as a measurable quality dimension — and how to operationalize it

Voice makes the need for that infrastructure especially concrete. Across GPT-4, Claude, Llama, and fine-tuned variants, identical prompts produce different voices. A production system cannot count on the same prompt to keep the same editorial character.

A July 2025 study that compares fine-tuning with system prompting gives evidence for this concern. It fine-tuned Llama 3.2-1B-Instruct using LoRA on a synthetically generated dataset derived from Wikipedia. Fine-tuning outperformed complex system prompting in achieving consistent conversational tone. The finding supports treating voice as a property shaped by training, reference material, and instructions at generation time.

The production approaches in the notes center on a curated brand corpus. Retrieval-augmented generation uses that corpus as the model writes, while fine-tuning makes the desired voice part of how the model behaves by default. A hybrid approach applies these methods to different risk surfaces. For founder-led content, the proposed corpus-building pattern starts with a 60-minute monologue, then extracts distinctive phrases and named opinions from its transcript. That material constrains drafting more concretely than a generic request to sound authoritative.

Measurement makes the constraint testable. BLEU/ROUGE comparisons against brand guidelines provide an overlap signal, while embedding cosine similarity to reference responses supports semantic drift detection. Human audit scorecards add editorial judgment, with Cohen’s kappa used to assess inter-rater agreement against a defined threshold. None of these measures establishes a universal definition of voice. Together, they make a team’s chosen definition explicit enough to evaluate and refine.

Building a practical quality framework that goes where formulas cannot

A rigorous framework treats writing as software in a specific sense. It defines requirements, tests outputs, routes failures for review, and treats iteration as part of production. Prose craft and voice coherence need model-based judges that work from clear criteria. Structural consistency calls for long-range coherence checks, while depth and impact call for human review or high-capability LLM rubrics. Each dimension needs an evaluation method suited to what it measures.

The resulting quality gate can use a cascade without confusing its layers. Formula-based checks provide an inexpensive first pass for surface extremes, including extreme length variance, but cannot certify coherence. Model-based judges assess the text in context, and humans review the truly hard cases. Adversarial review brings in independent challenges to find systematic failures that one evaluator may miss. Its value comes from testing the writing against requirements.

Those requirements give teams more to work with when they spell out failure states before they publish. Test cases and adversarial councils can look for filler patterns, voice drift, and structural collapse. Automated style enforcement then carries out the brand’s decisions the same way each time. A harness that flags “let’s delve,” “rich tapestry,” and “pivotal” is enforcing a chosen standard.

Provenance tracking belongs within the same infrastructure as quality tracking as AI detection and watermarking become production concerns in 2026. Attribution, however, does not establish whether a passage succeeds. The decisive standard remains whether its language is purposeful, its voice is coherent, and its structure converges on a clear result. Readability scores can contribute to that assessment, but they cannot stand in for it.