Chunking in RAG: why your splitter matters more than your embedding model
Most RAG retrieval problems I’ve debugged came down to the same thing: someone swapped the embedding model three times, added a reranker, then gave up — and never once changed the chunker.
This is backwards. The chunker decides what your embedding model is allowed to see. A great embedding on a bad chunk is still a bad retrieval. And the published research from the last 18 months keeps pointing at the same conclusion: the “smart” chunking strategies don’t beat a tuned dumb one. What does beat them is augmenting each chunk with context.
This post walks through the four chunking strategies you’ll actually run into, why semantic chunking disappoints on benchmarks, and a working contextual retrieval implementation with the numbers from Anthropic’s report. By the end you should have a default chunking recipe you can defend with data, not vibes.
The four chunking strategies
Almost every chunker in the wild is a variation of one of these.
1. Fixed-size
Split every N tokens (or characters) with some overlap.
def fixed_chunks(text: str, size: int = 512, overlap: int = 50):
tokens = text.split()
step = size - overlap
return [" ".join(tokens[i:i + size]) for i in range(0, len(tokens), step)]
Fast and reproducible. Cuts mid-sentence. Useful as a baseline so you have something to beat.
2. Recursive character splitting
The LangChain default. Tries paragraph breaks first, then sentences, then words — recursing until each chunk fits.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=100,
separators=["nn", "n", ". ", " ", ""],
)
chunks = splitter.split_text(document)
This is the pragmatic default for prose. It respects natural breaks when it can, falls back to character splits when it can’t.
3. Document-structure-aware
Uses the document’s own structure as the split signal — Markdown headers, HTML tags, code AST nodes. The chunks carry the section path as metadata, which is gold for filtering at retrieval time.
from langchain_text_splitters import MarkdownHeaderTextSplitter
splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
)
chunks = splitter.split_text(markdown_doc)
# each chunk's metadata: {"h1": "...", "h2": "...", "h3": "..."}
Use this whenever your source has structure. Throwing it away to run recursive character splitting is a self-inflicted wound.
4. Semantic chunking
Embed each sentence, walk through the document, and start a new chunk every time the cosine distance between adjacent sentences exceeds a percentile threshold.
from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding
splitter = SemanticSplitterNodeParser(
buffer_size=1,
breakpoint_percentile_threshold=95,
embed_model=OpenAIEmbedding(),
)
nodes = splitter.get_nodes_from_documents(documents)
Intuitively appealing. Topically coherent chunks should retrieve better. And it costs you an embedding call per sentence at index time.
The intuition is wrong often enough to matter.
Why semantic chunking often disappoints
Chroma Research ran a careful evaluation last year (Brandon Smith and Anton Troynikov, the latter a Chroma co-founder). They tested embedding-similarity splitters and LLM cluster chunkers against plain recursive and fixed-size chunking across multiple corpora, scoring with Intersection-over-Union and Recall.
The headline result: semantic methods produced inconsistent, often negligible gains. Sometimes they lost. Meanwhile they cost orders of magnitude more in embedding and LLM calls at index time.
The dominant variables across every experiment were chunk size and overlap, not the splitting strategy. A RecursiveCharacterTextSplitter at the right size was a hard-to-beat baseline.
If you’re going to spend engineering hours, spend them on a chunk-size sweep, not on a smarter splitter.
import numpy as np
def recall_at_k(retrieved_ids, relevant_ids, k=5):
return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(relevant_ids)
# Sweep chunk_size with everything else held constant
for size in [256, 400, 600, 800, 1000, 1200]:
chunks = chunk_corpus(documents, size=size, overlap=size // 8)
index = embed_and_index(chunks)
scores = [recall_at_k(index.search(q), gold) for q, gold in eval_set]
print(f"size={size} recall@5={np.mean(scores):.3f}")
You will see a clear curve, not a flat line. Pick the peak. Don’t ship a default you never measured.
What actually moves the needle: contextual retrieval
The interesting move isn’t a smarter splitter. It’s keeping the splitter dumb and giving each chunk back the context it lost when you split it.
This is Anthropic’s contextual retrieval recipe. For every chunk, prompt a cheap model with the full document and the chunk, and ask for 50-100 tokens of situating context. Prepend that context to the chunk before embedding.
import anthropic
client = anthropic.Anthropic()
CTX_PROMPT = """<document>
{doc}
</document>
Here is a chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Give a short (50-100 token) context that situates this chunk
within the overall document for retrieval. Answer only with the
succinct context."""
def contextualize(doc: str, chunk: str) -> str:
msg = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{
"role": "user",
"content": [
{"type": "text", "text": CTX_PROMPT.format(doc=doc, chunk=chunk),
"cache_control": {"type": "ephemeral"}},
],
}],
)
return msg.content[0].text
augmented_chunks = [
f"{contextualize(doc, c)}nn{c}" for c in chunks
]
The cache_control block matters. Without prompt caching you pay the full document token cost per chunk. With it, the document is cached once and reused across every chunk call — Anthropic reports roughly a 90% cost reduction on the context-generation step.
The reported numbers on their evaluation corpus (codebases, papers, fiction; top-20 retrieval failure rate):
- Contextual Embeddings alone: 35% fewer failed retrievals (5.7% → 3.7%)
- + Contextual BM25 (the same context augmentation applied to a BM25 index): 49% fewer (5.7% → 2.9%)
- + a reranker on top of both: 67% fewer (5.7% → 1.9%)
- One-time indexing cost: ~$1.02 per million document tokens with Haiku + prompt caching
- Optimal chunk size in their tests: 800 tokens with 100-token overlap, beating both 400 and 1600
The 800/100 number is worth pausing on. It’s not “256 because that’s what the tutorial said.” It’s not “1024 because the context window is big.” It’s a measured optimum on a real corpus. Yours will land somewhere similar but not identical — run the sweep.
When contextual retrieval pays for itself
Indexing cost goes up. Query-time cost is unchanged. So the math is:
- How often do you re-index? If you re-index weekly on a 100M-token corpus, that’s ~$100/week. Trivial for most production systems.
- What’s a retrieval miss worth? In support automation a single wrong answer can be measured in minutes of human time. The math is usually obvious.
Where it doesn’t pay: tiny corpora (< 1M tokens) where you can fit everything in context anyway, or extremely high-churn corpora where you re-embed many times a day. Everything else, run it.
Note: Contextual retrieval is additive with everything else. Recursive splitter, document-structure-aware metadata, BM25 hybrid, reranker — they all stack. The 67% number assumes the full stack. Don’t read that line as “the reranker is doing nothing.”
A default recipe to start from
If you’re staring at a blank file, this is a reasonable first pass:
- Recursive character splitter at 800 tokens, 100 overlap.
- Preserve any structural metadata (Markdown headers, file paths) as chunk metadata.
- Add 50-100 tokens of LLM-generated context per chunk with Haiku + prompt caching.
- Hybrid: vector index + BM25 over the same augmented chunks.
- Rerank top-20 down to top-5 with a cross-encoder.
- Build a 100-query eval set from real user logs and run a chunk-size sweep against your corpus before treating any of this as settled.
Step 6 is the one most teams skip. Don’t.
Wrapping up
Chunking is one of the highest-leverage things in a RAG pipeline and one of the least-measured. The cheap experiments — sweeping chunk size, adding contextual augmentation — usually beat the expensive ones (a fancier embedding model, a third reranker).
Two links worth your time next:
- Chroma’s chunking evaluation: https://research.trychroma.com/evaluating-chunking
- Anthropic’s contextual retrieval writeup: https://www.anthropic.com/news/contextual-retrieval
What chunk size do you run in production — and have you actually benchmarked it against alternatives, or is it still the framework default? I’m curious how often teams have a measured answer here.
Otomotiv Sanayii Derneği (OSD) 2026 yılının ilk 4 aylık dönemine ilişkin verileri açıkladı. Geçen yılın aynı dönemine göre toplam üretim yüzde 3 gerileyerek 448 bin 428 adet olarak gerçekleşti. Geçen yılın ilk 4 aylık dönemine göre yüzde 15 düşüş yaşanan otomobil üretimi ise 250 bin 276 adet oldu. Traktör üretimiyle birlikte toplam üretim ise 455 bin 526 adet olarak gerçekleşti. Ticari araç grubunda, yılın ilk 4 ayında üretim yüzde 17, ağır ticari araç grubunda yüzde 20 ve hafif ticari araç grubunda ise yüzde 17 artış gösterdi. 2025 yılının ilk 4 aylık dönemine göre ticari araç pazarı yüzde 6, hafif ticari araç pazarı ise yüzde 9 artarken ağır ticari araç pazarında yüzde 7’lik düşüş yaşandı. Yılın ilk 4 ayında bir önceki yılın aynı dönemine göre toplam otomotiv ihracatı adet bazında yüzde 9, otomobil ihracatı ise yüzde 27 geriledi. Bu dönemde, toplam ihracat 300 bin 718 adet, otomobil ihracatı ise 142 bin […]
Autocar tarafından aktarıldığı kadarıyla yeni jenerasyon BMW 1 Serisi, hem elektrikli hem de içten yanmalı motor seçenekleriyle 2028’de gelecek. Neue Klasse döneminin yeni tasarım dilini benimseyecek aracın, içten yanmalı tarafta 1.5 ile 2.0 litrelik dört silindirli benzinli motorlarla sunulacağı ve aynı zamanda plug-in hibrit seçeneğinin de listede olacağı söyleniyor. Burada 2.0 litrelik benzinli motor, elektrik motoru ve 100 km civarında tamamen elektrikli sürüş getirecek km 19,5 kWh kapasiteli bataryanın bir araya getirilmesi bekleniyor. Elektrikli tarafta tek bir elektrik motoru aracılığıyla 322 beygir güç sunması beklenen yeni araç için daha güçlü bir “M” versiyonunun hazırlanabileceği de aktarılıyor. Bu versiyonda gücün tam 463 beygire kadar taşınabileceği düşünülüyor. Tamamen elektrikli versiyonunda BMW i3’ten daha ufak batarya taşınacağı için 900 km menzil sunamayacak yeni 1 Serisi, kompakt bir seçenek olarak Avrupa başta olmak üzere birçok pazarda çok talep görebilir ve özellikle gençlerin ilgisini çekebilir.
Bugün Xiaomi’nin X hesabından yapılan paylaşıma göre “Xiaomi 17T” serisi resmi olarak 28 Mayıs’ta tanıtılacak. Kısa süre önce Roland Quandt tarafından detaylandırılan Xiaomi 17T, 6,59 inç boyutunda 120 Hz AMOLED ekran, MediaTek Dimensity 8500 Ultra işlemci, 12 GB LPDDR5X RAM, 256/512 GB UFS 4.1 depolama, 67W şarj destekli 6.500 mAh batarya ve 32 megapiksel selfie kamerasıyla geliyor. Cihazın arkasında ise 50 megapiksellik Light Fusion 800 sensörlü geniş açı, 5x optik zoom sunan 50 megapiksel telefoto ve 12 megapiksel ultra geniş açı kameralar bir araya getiriliyor. Üst model Xiaomi 17T Pro ise sızıntıya göre 6,83 inç boyutunda 144 Hz AMOLED ekran, MediaTek Dimensity 9500 işlemci, 12 GB LPDDR5X RAM, 512 GB UFS 4.1 depolama, 100W şarj destekli 7.000 mAh batarya ve 32 megapiksel selfie kamerasıyla geliyor. Cihazın arkasında 50 megapiksellik Light Fusion 950 sensörlü geniş açı, 5x optik zoom sunan 50 megapiksel telefoto ve 12 megapiksel ultra geniş açı kameralar bir […]
Bloomberg’den Mark Gurman tarafından aktarılan son bilgilere göre Apple, Gemini’den güç alacak yeni nesil Siri için ayrı bir uygulama sunacak. Bu uygulama aracılığıyla kullanıcılar Siri’yle, Gemini ya da ChatGPT ile olduğu gibi yazılı şekilde iletişim kurabilecek. Gurman’a göre bu uygulamada yapılan görüşmelerin bir süre sonra otomatik olarak silinmesine imkan tanıyan bir özellik de yer alıyor. İki yıldır beklenen yeni Siri’yi “beta” takısıyla kullanıma sunacağı aktarılan Apple, iddiaya göre sisteme güç verecek Gemini modellerini doğrudan kendi “Private Cloud Compute” sunucularında çalıştıracak. Bu sayede Apple kullanıcılarının yaptığı görüşmelere Google’ın erişim sağlayamayacağı söyleniyor. Resmi olarak önümüzdeki ay düzenlenecek WWDC26 etkinliğinde tanıtılacak yeni nesil Siri, genel kullanıma iOS 27 ve diğer yeni işletim sistemlerinin çıkacağı eylül ayında açılacak.