Chunking Strategies: The RAG Question Nobody Answers Well

Key takeaway: Ask ten candidates about chunking strategies for RAG and most will recite a list: fixed-size, recursive, semantic. The interviewer wants one sentence instead: "On my corpus, this chunker hit this recall, and here is why." No strategy wins everywhere, so the only good answer is a measured one. Build three chunkers, run them against one golden set, and keep the heading context attached to every chunk. The decision this post helps you make: which chunker to ship in your project, and how to defend it with a number.

Chunking looks like a preprocessing detail. It decides what your retriever can find and what your model gets to read. Plenty of RAG answers that look like model mistakes went wrong here, before the model saw a single token.

This post is one weekend build: three chunkers, one corpus, one eval loop, and the questions the interviewer asks once you show the results.

Previous post: Build a Chat-With-Your-Docs App That Survives a Doc Update.

What this post covers

• What chunking is, and why it exists at all
• Three chunking strategies, and what each one breaks
• The build: three chunkers, one corpus, one golden set
• What published benchmarks say about chunk size
• The questions the interviewer will ask, with answer skeletons
• A worked example and a five-minute demo script

Chunking is how a document becomes something you can retrieve

What it literally is. Chunking splits a document into passages before you embed them. Each passage gets one vector. At query time, the retriever compares the question's vector to every chunk's vector and returns the closest few.

Where it comes from. Embedding models accept a limited input length, and one vector for a 60-page PDF averages away almost everything specific in it. So you split. The chunk then does two jobs at once: it is the unit the retriever matches, and it is the unit the model reads.

Where it sits. Chunking is the middle of the ingestion pipeline: parse, chunk, embed, index. It is upstream of every retrieval trick you might add later, like hybrid search or reranking. Those can reorder what you have. They can't recover a fact that a bad split cut in half.

Technical explanation: small chunks give sharp vectors that match narrow questions, but lose the context around them. Large chunks keep context, but their vectors blur several topics together, and each one costs more tokens in the prompt.

Business explanation: picture an HR policy bot. The leave policy has a table: one row says "90 days", and the heading two lines above it says "Notice period, Grade 7 and above." A fixed-size split puts the heading in one chunk and the number in the next. An employee asks about their notice period, and the bot finds the number without the grade it applies to.

Three chunking strategies, and what each one breaks

1. Fixed-size. Cut every N tokens, with some overlap. It is cheap and predictable, and it ignores meaning. It will split a sentence, a table row, or a heading from its body without noticing.
2. Recursive. Try to split on paragraph breaks first, then lines, then spaces, until each piece fits. LangChain's recursive splitter uses ["\n\n", "\n", " ", ""] as its default separators, and measures size in characters unless you pass a token counter (docs). It keeps paragraphs whole, but still knows nothing about headings or tables.
3. Structure-aware. Split on the document's own structure: headings, sections, table boundaries, slide or page breaks. Prepend the heading path to each chunk so it carries its context. It needs a parser that sees structure, which is the hard part for PDFs.

Semantic chunking is the fourth option you'll be asked about. It embeds each sentence and splits where meaning shifts. It costs an embedding call per sentence at ingest, and a study by Qu, Tu and Bao found that "the computational costs associated with semantic chunking are not justified by consistent performance gains" over fixed-size chunks (arXiv 2410.13070). Expect the question, and have your reason for not defaulting to it.

The build: three chunkers, one corpus, one golden set

Scope. About 50 documents with real structure: policy handbooks, product docs, or a few annual reports. Mix prose, headings, and tables.

Golden set. 30 to 50 questions. For each, record the exact sentence or table row that answers it. Include some questions that need a heading to make sense ("notice period for Grade 7"), and some that need a table row.

The structure-aware chunker for Markdown or HTML-converted docs:

import re

def chunk_by_structure(md, max_tokens=400):
    sections, path, buf = [], [], []

    def flush():
        body = "\n".join(buf).strip()
        if body:
            sections.append((" > ".join(path), body))
        buf.clear()

    for line in md.splitlines():
        m = re.match(r"^(#{1,6})\s+(.*)", line)
        if m:
            flush()
            level = len(m.group(1))
            path[:] = path[:level - 1] + [m.group(2).strip()]
        else:
            buf.append(line)
    flush()

    chunks = []
    for heading, body in sections:
        for piece in recursive_split(body, max_tokens):  # big sections only
            chunks.append(f"{heading}\n\n{piece}")       # heading travels along
    return chunks

recursive_split is the recursive splitter from the previous section; any library version works. The sketch also treats # lines inside code fences as headings, so strip fenced code first if your docs have it.

One rule this sketch leaves out is tables: if a table is too long for one chunk, split it by rows and repeat the header row in every piece. Add that before you run the eval on documents with tables.

The eval loop:

chunkers = {"fixed_512": fixed_512, "recursive_400": recursive_400,
            "structure": chunk_by_structure}
for name, chunker in chunkers.items():
    chunks = [c for doc in docs for c in chunker(doc)]
    index = build_index(chunks)                       # same embedder for all
    hits = sum(any(gold in r.text for r in index.search(q, k=5))
               for q, gold in golden)
    print(name, f"recall@5={hits/len(golden):.2f}",
          f"chunks={len(chunks)}", f"avg_tokens={avg_tokens(chunks)}")

fixed_512, recursive_400, build_index and avg_tokens are thin wrappers over your splitter, vector store and tokenizer. Hold everything else constant: same parser, same embedding model, same k. Record chunk count and average tokens per chunk next to recall, because a bigger chunk can win recall and lose on prompt cost.

What published benchmarks say about chunk size

Use these numbers as reasons to run your own measurement.

• The best size depends on the questions. A GPU vendor's engineering team tested token chunks from 128 to 2,048 plus page-level and section-level chunking on five datasets. Factoid question sets did best with 256 to 512 tokens. Analytical sets over financial reports did best at 1,024 tokens or a page per chunk. Page-level had the highest average accuracy, 0.648, and the lowest variance across datasets (engineering blog, June 2025).
• Extremes lose. In the same tests, 128-token chunks scored 0.421 on one financial-filings set against 0.520 for page-level, and 2,048-token chunks scored below 1,024-token chunks on two others.
• A bare chunk loses its context. An AI lab's write-up on contextual retrieval uses a chunk that reads "The company's revenue grew by 3% over the previous quarter," with no company and no quarter. Prepending a short, chunk-specific context before embedding and BM25 indexing cut their top-20 retrieval failure rate by 49%, and by 67% with reranking (write-up). The heading path in the chunker above is a cheap version of the same idea.

What the interviewer will ask

1. "What chunk size did you use, and why?" Skeleton: the number, the corpus, the recall it got against the other two chunkers, and the cost per query. "I started at 512 tokens and measured three options" opens it well.
2. "How do you handle tables?" Skeleton: a table is one chunk if it fits. If it doesn't, split by rows and repeat the header row in each piece. Then show the golden-set question that proves it.
3. "Why not semantic chunking?" Skeleton: it adds an embedding call per sentence at ingest, and published results don't show a consistent gain. You would try it if structure-aware chunking failed on prose with no headings.
4. "What happens when you change the chunker?" Skeleton: every chunk ID changes, so you re-embed the corpus, rebuild the index, and re-run the golden set before switching traffic. Keep the old index until the new one passes.

Worked example: Karthik and the notice period table

Fictional scenario. Karthik is a backend engineer at a large IT services company, interviewing for an AI engineer role. He demos an HR policy bot built on a 40-page leave and exit policy.

The interviewer types: "What is the notice period for Grade 7?"

The bot answers "90 days" and cites a chunk that starts with Exit policy > Notice period > Grade 7 and above.

"Okay. What chunk size?"

"Sections, capped at 400 tokens, with the heading path on top," Karthik says. "I ran 512-token fixed chunks first. They split the notice period table from its heading. The bot kept answering 90 days for every grade, including interns. That question is in my golden set now."

"Why not just bigger chunks?"

"I tried 1,024. Recall went up, but every answer pulled in two extra pages of policy. Sections found it just as often at less than half the size."

Things to note: he answered with a measurement, he named the failure he found first, and he knew what the bigger chunks cost in prompt tokens.

How to demo it in five minutes

1. Show the corpus and one hard question that needs a heading or a table row (one minute).
2. Run the eval loop and show recall, chunk count, and average tokens for all three chunkers (one minute).
3. Show the chunk the fixed-size run returned for the hard question, with the heading cut off (one minute).
4. Show the structure-aware chunk for the same question, heading path on top (one minute).
5. Say which chunker you shipped and what would make you change it (one minute).

TLDR

• Chunking decides what the retriever can find and what the model reads, so it is upstream of every other RAG fix.
• Fixed-size is cheap and blind. Recursive keeps paragraphs. Structure-aware keeps headings and tables, and it is usually where you end up for structured documents.
• Published benchmarks disagree on the best size because the best size depends on your questions. Measure on your own golden set.
• Attach the heading path to every chunk. It is a cheap way to give a bare chunk its context back.
• In the interview, answer with a number you measured on your own corpus.

Where to start

Write the golden set first, including five questions that need a heading or a table row. Then build the three chunkers and the eval loop, and only then pick a size.

This is Day 4 of the 30-day series on interview prep and build guides for AI app builders. Follow Flow on X for the rest. Doing structure-aware parsing and re-chunking across a dozen messy sources at work? Inherent's context engine runs ingestion as a managed step, with a receipt that shows which chunk produced each answer. DM for a design-partner slot.

Next read: RAG Architecture Tradeoffs in Plain English