About this project
The hypothesis, experiment framework, and how the scoring works.
The hypothesis
Documentation is written for human readers, but increasingly it's also being read by AI systems — chatbots, RAG-powered assistants, LLM agents that answer questions using docs as a knowledge source. Writing guidelines evolved for humans: use active voice, define your terms, be specific. Do those same rules matter for AI readers too?
My hypothesis going in: yes, but not uniformly. I suspected vague writing would hurt AI badly (AI can't fill in what isn't there), passive voice would also be a significant problem, and undefined jargon might have an asymmetric penalty depending on how the doc is accessed.
The experiments confirmed most of this — and found one failure mode I didn't fully anticipate: pronoun ambiguity doesn't just confuse AI, it produces confident wrong answers. They also found one non-finding: passive voice has no measurable effect on AI accuracy.
The three experiments
Writing quality
Five degradation types introduced one at a time into two synthetic documents. Eight AI models across three providers tested in both full-document and RAG modes.
RAG strategies & embedders
Five retrieval strategies and five embedding models tested independently and in combination on the complex document. Uses the same writing-quality corpus with a subset of models.
Semantic markup & chunking
A new multi-topic API reference document, nine document variants (markup stripping, generic labels, overlapping vocabulary), two chunking strategies (heading-based vs. fixed-size), four models, two embedding models.
All experiments use the same framework: write or degrade a document, run a fixed question set, score automatically, compare across variables. Each experiment page describes its specific design.
The documents
The experiments use three synthetic technical documents designed to resemble real API documentation:
API Authentication Guide
~400 words. Single topic: how to authenticate API requests using a key. One credential type, straightforward instructions.
SDK Integration Guide
~800 words. Multiple topics across sections: two credential types, rate limiting, error handling, key rotation. Facts spread across the document.
Acme Platform API Reference
~1,000 words. Nine sections covering authentication, OAuth, pagination, rate limiting, webhooks, error responses, and versioning. Multiple overlapping vocabulary variants.
Each document was written in a clean "control" version first, then degraded in specific ways — one variable at a time so any accuracy change can be attributed to that variable.
Degradation types
Experiments 1 & 2 — five writing quality degradations:
"Every request must include the key" → "The key must be included in every request." No information lost — just restructured.
Replaced specific nouns with pronouns when two different things could be "it." The document had two credentials; both became "it."
Replaced all meaningful terms with made-up abbreviations — "API key" became "BT," "Authorization header" became "AC-H." The information exists but is encoded in vocabulary that only appears in this document.
"Rotate every 90 days" → "rotate periodically, roughly quarterly or so." The specific value is replaced with a vague approximation.
"Every request must include X" → "requests should generally include X where applicable." The instruction becomes qualified and uncertain.
Experiment 3 — three structural degradations, plus overlapping vocabulary variants:
All markdown formatting removed — headers, lists, code blocks, and tables converted to flat prose paragraphs. Same factual content, zero structure.
Full structure preserved, but every heading replaced with "Overview", "Section 1" through "Section 10". Formatting intact, labels meaningless.
Sections rewritten so every part shares credential vocabulary. Factual content is still unique per section, but surrounding language is not — mimicking badly written docs where every section sounds the same.
The retrieval modes
This is worth explaining clearly because the naming is confusing. There are two completely different things called "context" in this project, plus a third mode that's different from both:
Full-doc mode (called "Context" in the results)
The entire document is placed directly into the AI's prompt. No search, no chunking. The AI reads everything at once. This is how you'd use AI if you just pasted a doc into a chat window.
This is the upper-bound test — if the AI fails here, the writing is the problem. If the AI succeeds here but fails in RAG mode, the retrieval is the problem.
Contextual retrieval (a RAG strategy, not a mode)
Before embedding each chunk for search, an AI writes one sentence describing how that chunk fits into the broader document. That sentence is prepended to the chunk before embedding. This gives the search index document-level context it would otherwise lack.
This is not the same as giving the AI the full document. The AI still only sees a few retrieved chunks — the context sentence just makes the embedding smarter.
Agent retrieval (called "Agent RAG" in the results)
Instead of a single fixed search, the AI receives a search(query) tool it can call up to four times per question. It issues a query, reads the results, and can try different phrasings or follow-up searches before answering.
Most deployed doc assistants are not agentic — they do one fixed pass. Agent retrieval is useful in these experiments as a diagnostic: it reveals which writing failures are structurally recoverable versus which aren't.
In the results table, "Context" means full-doc mode. "Contextual" means the contextual retrieval RAG strategy. "Agent RAG" is the agentic mode. They are completely different things that happen to share words.
The RAG pipeline
In RAG mode, the document is split into small chunks. When a question comes in, the system finds the most relevant chunks using semantic search — both the question and the chunks are converted to numerical vectors by an embedding model, and the top-3 most similar chunks are retrieved. The AI answers using only those chunks; it never sees the full document.
This pipeline has three knobs tested across the experiments:
- Retrieval strategy — how the search is structured (naive lookup vs. HyDE, multi-query, parent-document, or contextual retrieval). Tested in Experiment 2 with the embedding model held constant.
- Embedding model — what turns text into searchable vectors. Tested independently in Experiment 2 with the strategy held constant (naive RAG).
- Chunking strategy — how the document is split. Experiment 3 compares heading-based chunking (splits at section boundaries) versus fixed-size character chunking (splits every 400 characters regardless of structure). Most production RAG pipelines that ingest web-crawled content use fixed-size chunking.
Running these as separate experiments lets the results isolate what each variable actually contributes. See RAG strategy and embedder comparisons → · See chunking strategy comparisons →
Embedding models tested
Embedding models are not the same as the AI models above. They don't generate answers — they convert text to numerical vectors so the search step can find relevant chunks. Think of them as the search index, not the answerer.
Five embedding models were compared across three providers in Experiment 2 using naive RAG on the complex document. Experiment 3 compared OpenAI small and Voyage 3 on the semantic markup document with fixed-size chunking.
- text-embedding-3-small
- text-embedding-3-large
Small is the baseline for all strategy comparisons. Large gains +7pp on pronoun ambiguity. Caution: combining OpenAI large with parent-document retrieval reverses its coreference benefit. In Experiment 3 with fixed chunking, OpenAI small fails on overlapping vocabulary — Voyage recovers the gap.
- gemini-embedding-001
Modest gains (~4–6pp) over OpenAI small across most degradation types on the complex document.
- voyage-code-3
- voyage-3.5
- voyage-3
Voyage 3.5 is the best all-rounder across Experiments 1 and 2 — +10pp on jargon for haiku, consistent gains on coreference. Voyage 3 eliminates the overlapping-vocabulary failure mode in Experiment 3 (fixed chunking: 85% → 100%).
Scoring
Each AI response was scored two ways:
Fact matching
Each question has a set of expected facts (specific values, terms, or phrases) that a correct answer should contain. If the AI's answer includes those facts, the question passes. The overall score for a run is the percentage of questions that pass.
This is objective and consistent, but it can miss nuance — a vague-but-passing answer looks the same as a precise one.
Rubric scoring
A second AI (claude-haiku-4-5-20251001) rates each answer on four dimensions: factual accuracy, specificity, completeness, and whether it introduced anything false. Each dimension is 0–2; total is 0–8.
This catches failures fact matching misses — like an answer that echoes vague source language and sounds reasonable but contains no useful information.
The results tables show fact-match percentages. Rubric scores are shown where they reveal something the percentages don't.
AI models tested
These are the large language models that read the documents and generate answers — the subjects of the experiment. Eight models across three providers were tested on the base battery (Experiments 1 & 2); a subset was used for Experiment 3.
- claude-haiku-4-5-20251001
- claude-sonnet-4-6
- claude-opus-4-8
- gpt-4o-mini
- gpt-4o
- gpt-5.5
- gemini-2.5-flash
- gemini-3.5-flash
For the RAG strategy, embedder, and chunking comparisons, three representative models were used: claude-haiku-4-5-20251001, gpt-4o-mini, and gemini-2.5-flash.
Limitations
- Synthetic corpus. All documents were written specifically for these experiments. Real docs have messier, more varied degradation patterns.
- Small question sets. 8–10 questions per document version. Scores should be read as directional, not precise measurements.
- RAG strategy comparisons use a single embedding model. The five-strategy comparison all use OpenAI's
text-embedding-3-small— a different embedder might shift the relative strategy rankings. - Automated scoring. Fact matching is objective; rubric scoring uses an AI judge, which is consistent but not human-validated.
- Not peer-reviewed. This is a personal project. The methodology is transparent and the data is open, but it hasn't been independently replicated.