Bad writing is bad for AI and human readers. Better models don't fix it.
Experiments on synthetic docs — degrading one writing quality at a time, then testing eight AI models across three providers.
Vague writing is a hard ceiling — and the two types work differently
Two types of vague writing destroyed accuracy across every model and provider tested. Hedged instructions — "requests should generally include X where applicable" instead of "every request must include X" — and vague quantities — "rotate periodically, roughly quarterly" instead of "rotate every 90 days." On the complex document with RAG retrieval, both dropped scores from ~97% to 48–61%. Claude Opus and Claude Haiku scored identically. GPT-5.5 matched GPT-4o. The writing became the ceiling and no model from any provider could get above it.
They're different failure modes, though. Hedged instructions are partially recoverable with better retrieval — the information is in the document, just qualified. Vague quantities are not: when "every 90 days" was replaced with "periodically," the specific value disappeared from the document entirely. No retrieval approach can recover it, and models will synthesize plausible-sounding values from training data to fill the gap — stated confidently, not flagged as a guess.
Pronoun ambiguity produces confident wrong answers
When "it" could mean either of two credentials, AI doesn't say "I'm not sure." It picks one confidently and describes it in detail — wrong, with authority. Scores dropped to 61–76% on the complex document. This is the sneakiest failure mode: when AI hedges, a user knows to look elsewhere. When it gives a confident wrong answer, they act on it. Ambiguous pronouns in docs produce authoritative misinformation, which is a different and worse outcome than a retrieval miss.
Undefined jargon breaks RAG in two separate ways
When docs use unexplained abbreviations, there are two distinct failures at once: the AI can't understand the content and can't find the right section in the first place. Search works by matching meaning — if your doc says "BT" and someone asks "how do I authenticate," the embedding search doesn't connect them. With the full document in the prompt, models scored 74–87%, using general knowledge to infer what abbreviations probably mean. In RAG mode, scores fell to 59–70%. The content existed but was invisible to the retrieval step.
| Writing problem | Score | Drop | What users experience |
|---|---|---|---|
| Clean doc (baseline) | 100% | — | — |
| Hedged instructions | 54% | −46pp | Hedged answer — users know they didn't get a real answer |
| Vague quantities | 56% | −44pp | Confident wrong answer — model synthesizes a plausible value |
| Undefined jargon | 65% | −35pp | Silent failure — relevant section never retrieved |
| Pronoun ambiguity | 69% | −31pp | Confident wrong answer — model commits to one reading of "it" |
| Passive voice | 93% | −7pp | Nothing significant |
Can better retrieval infrastructure help?
Partly — and the limits reveal something important about which writing problems are recoverable and which aren't.
Embedding models make a meaningful difference for jargon and pronoun ambiguity, but the gains are model-dependent. Voyage 3.5 improved undefined jargon by +10pp for claude-haiku, but only +7pp for gemini-2.5-flash and +4pp for gpt-4o-mini. OpenAI large's +7pp coreference benefit holds consistently across all three models — but it actively hurts jargon retrieval for gpt-4o-mini (−10pp) and gemini-2.5-flash (−6pp). On vague writing, all embedders hit the same floor regardless of model. A better embedder can only retrieve information that exists.
Agent retrieval — where the LLM controls a search() tool it can call repeatedly — reaches the best results in the dataset for claude-haiku: 95% on undefined jargon (+30pp over naive RAG), 87% on pronoun ambiguity (+18pp). Gains are haiku-specific on the complex document: gemini-2.5-flash regresses 4pp on jargon with agent retrieval, while gpt-4o-mini gains only 6pp. Even on the best model, vague quantities only improve 2pp — the specific value was never written down, so no number of searches helps.
Two more things the experiments measured
Document complexity amplifies every problem. The same writing issues hurt significantly more in a longer, multi-section document than in a short single-topic one. Hedged instructions dropped scores 16pp on the short doc — but 39pp on the complex one. Short docs give AI enough nearby context to partially compensate; multi-section docs with facts distributed across chunks don't.
Passive voice doesn't matter. Every model scored 85–100% on passive-voice versions across all three providers. AI doesn't need a named actor to understand an instruction the way human readers do — this turned out to be a non-finding. One less thing to worry about.
The bottom line
| Writing problem | Severity | Where it hits hardest |
|---|---|---|
| Vague quantities ("periodically") | Very bad — no fix | Model synthesizes a plausible value — users get a confident wrong answer they can't detect |
| Pronoun ambiguity ("it", "this") | Bad | Model commits to one interpretation — users get a confident wrong answer |
| Undefined terms / jargon | Bad | Silent failure in most systems — relevant section never retrieved; 3–4× call cost in agentic systems |
| Hedged instructions ("where applicable") | Bad — partial fix exists | Users get hedged answers they know they can't act on; agent retrieval recovers +19pp |
| Passive voice | Barely matters | No meaningful degradation for AI — matters for human readers, not models |
Bigger, newer, or more expensive models will not compensate for vague writing. Neither will switching providers or using a smarter RAG strategy. The writing is the bottleneck.
Personal project by CT Smith, technical writer — not peer-reviewed research. Synthetic corpus, small query sets, automated scoring. Results are directionally interesting, not statistically rigorous. Full limitations →