Semantic Markup & Chunking Strategy

The short answer

Markup does not affect RAG accuracy when using heading-based chunking. Even on a long, multi-topic document with overlapping vocabulary across sections, removing all formatting or replacing every heading with "Section 1–10" produced no accuracy loss. Body text uniqueness — specific technical terms like HMAC-SHA256 or grant_type=refresh_token — is what drives retrieval.

Markup does affect accuracy when using fixed-size character chunking and the content has overlapping vocabulary. That combination is common in production: most RAG pipelines that ingest web-crawled content use fixed-size chunking, and technical documentation often shares vocabulary across authentication, error handling, and security sections. Under those conditions, the experiment design that produced good headers but overlapping content scored 85% (vs. 100% with heading-based chunking) — a 15-point gap confirmed across four models.

The finding is not about whether embedding models "understand" headings. It's about chunk boundary alignment: heading-based chunking keeps section content together, so retrieval can find the right section. Fixed-size chunking crosses boundaries, and when sections share vocabulary, chunks from the wrong section get retrieved.

Experiment design

The experiment used a ~1,000-word multi-topic API reference (authentication, OAuth, pagination, rate limiting, webhooks, error responses, versioning) and 10 queries targeting specific facts in specific sections. Nine document versions were tested:

Two chunking strategies were tested using the same underlying embedding model (OpenAI text-embedding-3-small):

• Heading-based: splits at h1–h3 boundaries, producing one chunk per section (~111 chars/chunk average). k=3 retrieves ~33% of the document.
• Fixed-size: splits every 400 characters (50-character overlap, word-boundary snapping), ignoring document structure entirely. Produces 21 chunks; k=3 retrieves ~14% of content.

Round 1 — ceiling effect on short docs

The first version used a ~400-word authentication guide with 5 sections. Result: 100% across every version, every embedder. Not because markup doesn't matter — because a 5-section document at k=3 returns 60% of the doc on every query. There aren't enough chunks to miss the right one.

Conclusion: short single-topic documents are immune to retrieval degradation regardless of markup or chunking strategy. A meaningful test requires a document long enough that retrieved chunks are a small fraction of total content.

Round 2 — longer doc, heading-based chunker

A ~1,000-word document with 9 sections: with heading-based chunking and k=3, retrieval covers ~33% of the document. The prediction: removing headings should hurt accuracy because the embedding can no longer use heading text to anchor retrieval.

Result: still 100% across every variant.

Why? Body text uniqueness drives retrieval. Even with generic headings ("Section 7"), the section body contains vocabulary that only appears in that section: webhooks:manage in the Scopes section, grant_type=refresh_token in the OAuth section, HMAC-SHA256 in the Webhooks section. The embedder doesn't need the heading to find the right chunk — the facts themselves are unique enough.

Heading labels carry zero marginal retrieval value when section body text is topically distinct. The chunker does all the work: by respecting section boundaries, it ensures each chunk contains one section's content and nothing else.

Round 3 — fixed-size chunking shows the signal

Most production RAG pipelines ingest content from crawlers that don't preserve document structure. They use fixed-size character chunking (300–512 tokens) regardless of section boundaries. With 400-character chunks, this experiment produces 21 chunks from the same document; k=3 retrieves ~14% — a far more competitive selection problem.

This is where markup finally matters — but only for the overlapping-content versions.

What broke: The overlapping-content version with good headers dropped from 100% to 85%. The clearest single failure: q4 (webhook scope lookup) scored 0%. The webhooks:manage scope row ended up in a chunk half-occupied by credential vocabulary from an adjacent section — so the embedder retrieved that chunk for the wrong reason (credential language) and missed the scope table entirely.

The −2pp on clean versions (original, stripped, generic-labels) is incidental: the rate-limit table (q6) straddles a chunk boundary, so the Growth plan row lands in a chunk without enough context to answer confidently. That's a fixed-chunking cost on any document, not a markup effect.

Cross-model validation

The 85% result from claude-sonnet is confirmed across four models. All show the same pattern: markup stripping alone has minimal effect; overlapping vocabulary combined with fixed chunking causes consistent degradation. Click any score to see the underlying responses.

The pattern is consistent across providers: markup stripping has near-zero effect (±3pp), overlapping + fixed chunking causes a 7–13pp drop. Gemini is the most affected (−13pp on overlapping); sonnet and gpt-4o-mini tie at −13pp and −9pp respectively. No model is immune.

Embedding model comparison (fixed chunking)

A separate run used Voyage AI's voyage-3 embedding model in place of OpenAI text-embedding-3-small, with the same fixed-size chunker and claude-sonnet as the agent. Voyage is optimized for code and technical content — the hypothesis was that better technical vocabulary representations might reduce the overlapping-content retrieval errors.

Voyage's embeddings recover the full 15pp gap on the overlapping-content version — the case that dropped OpenAI small to 85% scores 100% with Voyage. This is the same pattern observed in earlier experiments where Voyage's technical vocabulary training produces more widely-spaced embeddings for semantically distinct but lexically similar content. When overlapping credential vocabulary appears in chunks from different sections, Voyage's embedding space is still able to tell them apart; OpenAI small is not.

The finding narrows the problem: the overlapping + fixed-chunking failure is partly an OpenAI small embedder weakness, not a universal property of fixed-size chunking. A better embedder (Voyage 3, and likely OpenAI large) reduces the semantic confusion that overlapping vocabulary creates. The writing quality problem — using the same vocabulary across sections — still exists and still costs you something, but the magnitude depends on your embedding model.

Why stripped overlapping recovers relative to headed overlapping

The overlapping + stripped version (95%) scores better than overlapping + good headers (85%) across every model. This is counterintuitive — stripping the headings from an already-problematic document makes it more accurate, not less.

The mechanism: with fixed-size chunking, heading tokens create density spikes. A section heading like "## Scopes and Permissions" adds ~4 tokens to the start of a chunk, pushing the following table rows toward a chunk boundary. With the heading removed, the chunker distributes tokens more uniformly across the content — the scope table lands in a cleaner chunk without heading tokens from an adjacent section consuming part of it.

This is not a general argument for removing headings. With heading-based chunking, descriptive headings are completely irrelevant to accuracy (because the chunker already respects section boundaries). With fixed-size chunking and clean vocabulary, headings are also irrelevant. The counter-intuitive recovery only appears in the specific intersection: fixed-size chunking + overlapping vocabulary + section headings adding density at boundaries.

What this means for documentation

If your RAG pipeline uses heading-based or semantic chunking: markdown structure is for human readers, not retrieval. Removing headers, stripping lists, or genericizing heading text has no measurable accuracy effect. Body text uniqueness — specific values, identifiers, parameter names — is all that matters for retrieval.

If your pipeline uses fixed-size character chunking (most production systems that ingest web-crawled or unstructured content): clear, topically distinct sections reduce the risk that chunk boundaries cut through the middle of critical content. Write focused sections where each part uses vocabulary specific to that section's purpose. The fix is primarily a writing-quality issue — but in that context, section distinctiveness is load-bearing for chunking, not just for readability.

The failure condition that breaks everything regardless of chunking strategy: vague writing that removes specific facts entirely. "Rotate periodically" instead of "every 90 days." No amount of heading structure or chunking strategy recovers information that was never written down. See the main findings →