RAG Strategies & Embedding Models

Results (claude-haiku, api-sdk-complex)

Scores are average percentage of questions answered correctly. Bold = best for that row. Click any score to see the underlying AI responses.

Contextual retrieval — confirmed across 3 providers

The coreference finding wasn't a fluke. Every provider tested showed worse scores with contextual retrieval on the coreference-degraded document.

Parent-document — confirmed across 3 providers

The undefined-terms improvement from parent-document retrieval held across all three providers.

Embedding model comparison — complex doc (naive RAG)

The embedding model is a separate variable from retrieval strategy. Five models were first compared using naive RAG on the complex document with claude-haiku. Two results stood out:

OpenAI large leads on pronoun ambiguity (+7pp) — higher-dimensional representations handle "it could mean either credential" better. Both Voyage models lead on undefined jargon (+10pp over OpenAI small) — their technical training data handles abbreviations that look like code identifiers.

Those haiku findings were then cross-validated with gpt-4o-mini and gemini-2.5-flash on the two most informative embedders:

Voyage 3.5 jargon gains are model-dependent

Voyage 3.5 improves jargon retrieval across all three models, but the magnitude varies significantly — from +10pp for haiku down to +4pp for gpt-4o-mini. The coreference benefit (+3–4pp) is more consistent. The large haiku gain shouldn't be assumed to generalize.

OpenAI large hurts jargon for gpt-4o-mini and gemini

OpenAI large's precision embeddings help haiku retrieve more relevant jargon chunks — but actively hurt gpt-4o-mini and gemini. This isn't just the parent-doc interaction found earlier; it's the embedder itself conflicting with how these models process retrieved content. OpenAI large's coreference benefit (+5–6pp) held consistently across all three models. It is a precision tool: good for disambiguation, risky for vocabulary coverage.

The vague writing floor holds across all embedders and all models (54–61% on the complex document). No embedder recovers information that isn't in the document.

Embedding model comparison — short doc (naive RAG · claude-haiku)

The same three embedders tested on the short document tell a different story about vague writing:

On the complex document, all five embedders bottomed out at 54–61% on vague writing — a hard floor. On the short document, both Voyage 3.5 and OpenAI large gain +5pp on quantifier vagueness over OpenAI small. The floor is a complexity effect, not an inherent ceiling. A short, single-topic document gives the LLM enough nearby context to partially recover from missing specifics. The complex document — with facts distributed across sections — doesn't. OpenAI large's +7pp coreference benefit from the complex doc doesn't appear here; the short doc has only one credential type, so pronoun ambiguity is a smaller problem.

Strategy × embedder: they don't always stack — and sometimes conflict

Independently optimizing retrieval strategy and embedding model, then combining them, does not reliably produce the expected gains.

Parent-doc + Voyage 3.5: neutral on jargon, complementary elsewhere

Parent-document retrieval (+20pp on undefined terms) and Voyage 3.5 (+10pp on undefined terms) seem like they should be complementary. They aren't — at least not for jargon:

The +1pp on undefined terms is effectively nothing. Parent-doc fixes jargon by giving the LLM more context at generation time — once the full parent section is in the prompt, the model can infer abbreviation meanings from surrounding sentences. Voyage 3.5 fixes jargon by improving retrieval matching. When you already have the larger context window, better retrieval adds almost no marginal value for jargon.

Where they are complementary: clean doc hits 100% (+8pp) and pronoun ambiguity improves +5pp. Those failure modes benefit from both better retrieval and more context — they aren't addressing the same problem.

Parent-doc + OpenAI large: the benefit reverses ⚠

OpenAI large gains +7pp on coreference in naive RAG. Parent-doc is strategy-neutral on coreference. Combining them was expected to at least preserve OpenAI large's coreference benefit — instead it erases it:

Coreference: OpenAI large's +7pp advantage from naive RAG disappears with parent-doc (76% → 68%). In naive RAG, higher-dimensional embeddings retrieve a precise chunk that helps disambiguate pronouns. With parent-doc, retrieval expands to a full section — which introduces more ambiguous pronoun references rather than fewer.

Undefined jargon: Parent-doc + OpenAI large (75%) is −10pp worse than parent-doc + OpenAI small (85%). OpenAI large's precision retrieves the "closest match" parent section — which may not be the section containing the abbreviation definitions. OpenAI small's fuzzier matching casts a wider net that more often captures those definitions.

The takeaway: embedder precision and context expansion are in tension for some failure modes. The same property that makes OpenAI large better at naive RAG (precision) makes it worse than OpenAI small when paired with a strategy that depends on broader context retrieval. Optimizing strategy and embedder separately, then combining, does not produce the expected sum.

What the agent experiment reveals about retrieval failure

Most deployed RAG systems are not agentic. The majority do a single fixed retrieval pass and answer from whatever came back. The agent experiment is useful not primarily as a deployment target, but as a diagnostic: it separates which failures are structurally recoverable from which aren't. That distinction matters because it determines what users actually experience when retrieval goes wrong.

Note: the haiku results that follow are striking, but cross-validation with two other models shows the gains are haiku-specific — gemini-2.5-flash actually regresses on jargon with agent retrieval. See cross-validation below.

What users experience when fixed RAG fails

Accuracy scores by mode (claude-haiku · api-sdk-complex)

The numbers behind the table above. Agent RAG used a search(query) tool callable up to four times per question — it issues a query, reads results, and can refine its phrasing before answering.

The 95% on undefined jargon is the best result in this entire dataset for claude-haiku — better than parent-doc + Voyage 3.5 (86%), better than any fixed strategy or embedder combination. The agent discovers vocabulary iteratively: it searches for an unknown abbreviation, finds a chunk that uses it in context, then searches again for what to do with it. It never needs to match the jargon on the first try.

Agent retrieval gains are haiku-specific

Cross-validation with gpt-4o-mini and gemini-2.5-flash reveals that haiku's gains do not generalize. The jargon improvement — the headline result above — is driven almost entirely by one model:

Gemini regresses. This suggests the jargon recovery mechanism — iterative vocabulary inference across multiple search calls — is haiku-specific behavior, not a property of agentic retrieval in general. Haiku's smaller context window may force tighter, more targeted queries that happen to work well for abbreviation lookup. Gemini's larger context window may lead it to retrieve noisier chunks across its multiple passes, compounding vocabulary mismatches rather than resolving them.

The coreference (+18pp) and scope hedging (+19pp) haiku results also need cross-validation before generalizing. The finding that vague quantities are unrecoverable (+2pp) held consistently across all three models and does generalize.

Confabulation is the hidden risk

Vague quantities are the highest-priority documentation problem — not because of the accuracy score, but because the failure mode is invisible to users. A model that can't find a specific rotation interval doesn't say "I don't know." It says "rotate every 30 days" with the same confidence it would use if that number were actually in the doc. Users have no signal that they received a synthesized answer rather than a retrieved one.

Scope hedging fails differently. Users get "it generally depends on your setup" — an answer they know they can't act on. Frustrating, but visible. Confabulated values are actionable-seeming and wrong.

The two vague writing types are not the same problem

Agentic retrieval has real costs

Even when agent retrieval works, it is not free. Undefined jargon required 3–4 search calls per question — roughly 3–4× the latency and budget of a single-pass system. On a clean document, agent mode scores 7pp lower than naive RAG (93% vs 100%), because iterative searching introduces noise: the agent occasionally retrieves suboptimal chunks across its multiple passes.

The implication for production systems: agent retrieval solves vocabulary mismatch well, but the per-query cost scales directly with how often your documents use undefined jargon. That cost is avoidable — it is documentation debt made visible as infrastructure overhead.

Priority order for documentation writers