Does writing quality affect AI accuracy?
A controlled-degradation experiment framework — degrade one variable at a time, measure what breaks. Three experiment sets so far.
Writing quality
Five degradation types — passive voice, pronoun ambiguity, undefined jargon, vague quantities, hedged instructions — tested across eight models and three providers. Both full-document and RAG retrieval modes.
Finding: Vague writing creates a hard accuracy ceiling that no model, provider, or retrieval strategy can overcome. The two vague writing types fail differently — one is unrecoverable, one is partially fixable. Passive voice has no measurable effect.
See findings →RAG strategies & embedders
Five retrieval strategies (naive RAG, HyDE, multi-query, parent-document, contextual) and five embedding models (OpenAI, Google, Voyage) tested independently and in combination on the same document.
Finding: Strategy matters more than embedder. Agent retrieval tops the dataset but gains are model-specific — gemini-2.5-flash regresses on jargon. Combining a good strategy with a good embedder does not reliably stack; they can conflict.
See findings →Semantic markup & chunking
Does removing markdown structure — headers, lists, code blocks — degrade RAG accuracy? Tested across three document variants, two chunking strategies, four models, and two embedding models.
Finding: Markup has no effect with heading-based chunking. With fixed-size chunking and overlapping vocabulary across sections, it causes a 15pp drop — but only with OpenAI small embeddings. Voyage 3 eliminates the gap entirely.
See findings →Personal project by CT Smith, technical writer — not peer-reviewed research. Synthetic corpus, small query sets, automated scoring. Results are directionally interesting, not statistically rigorous. Full limitations →