Context engineering: why your RAG breaks at 50k tokens and how to fix it

TL;DR

Naive RAG works in the demo and breaks in production. The pattern that has stabilized in 2026 combines seven techniques: Chonkie for semantic chunking (not character-based), Qdrant with hybrid dense + BM25 search, self-hosted BGE-reranker v2-m3 to rerank top-k, Contextual Retrieval (Anthropic's technique) which prepends context to each chunk via Haiku in batch, RAPTOR for hierarchical summaries, RedisVL as semantic cache, and continuous evaluation with Ragas + DeepEval traced in Langfuse. This article shows the end-to-end pipeline with code, real metrics (recall@5 rises from 0.62 to 0.91) and costs at scale. The point where "your RAG breaks" is almost always chunking + missing reranker.

The symptoms

The agent worked perfectly with 20 documents in QA. It went to production with 500 and started to:

• Fabricate details not in the source.
• Retrieve the right chunks but incomplete.
• Ignore recent docs because old ones have a stronger embedding.
• Reply "no information found" when the answer was there.
• Mix information from two different clients.

Each symptom has a distinct cause, and each cause has a solution.

The pipeline in 7 steps

  Raw document
       │
       ▼
  [1] Parsing with Unstructured.io or Firecrawl
       │
       ▼
  [2] Semantic chunking (Chonkie)
       │
       ▼
  [3] Contextual Retrieval: prepend context to each chunk
       │
       ▼
  [4] Dense embedding (OpenAI 3-small or BGE-m3) + BM25
       │
       ▼
  [5] Qdrant (hybrid collection)
       │
       ▼
  Query time:
       [6] Hybrid search (dense + sparse) → top-40
       ▼
       [7] Rerank with BGE-reranker → top-6
       ▼
       LLM gets clean context

Step 1 — Serious parsing

Forget PyPDF2. Unstructured.io extracts with structure: it distinguishes headers, tables, code, lists. Firecrawl does the equivalent for web pages, handling JS and pagination.

from unstructured.partition.auto import partition

elements = partition("sat-tax-manual-2026.pdf")
for el in elements:
    print(el.category, "|", el.text[:80])
# → Title, NarrativeText, Table, ListItem, FigureCaption, ...

Result: each chunk knows its content type. Tables stay as tables (not split in half), headers link to their paragraphs.

Step 2 — Semantic chunking with Chonkie

LangChain's RecursiveCharacterTextSplitter cuts every 1,000 characters. It ends up splitting an important table in half or chopping a sentence midway.

Chonkie has three better strategies:

• SemanticChunker — splits where embedding similarity between consecutive sentences drops.
• SDPMChunker — Semantic Double-Pass Merging: initial semantic chunks, then merge to respect target size.
• LateChunker — embed the entire document first, then chunk. Preserves long-range context.

from chonkie import SDPMChunker

chunker = SDPMChunker(
    embedding_model="BAAI/bge-m3",
    chunk_size=512,
    threshold=0.75,
    skip_window=1,
)

chunks = chunker.chunk(document_text)

For manuals/contracts, SDPMChunker works best. For short tickets/emails, SemanticChunker.

Step 3 — Contextual Retrieval

Technique published by Anthropic in 2024, now standard. Problem: a chunk that says "the deadline is 30 days" is ambiguous without knowing which deadline it refers to.

Solution: before embedding, prepend to the chunk a context paragraph that the document gives it. Done once per chunk with a cheap model (Claude Haiku in batch API, cost ~$0.03 per 1,000 chunks).

CONTEXT_PROMPT = """Full document:
<document>
{document}
</document>

Here is the chunk we will index:
<chunk>
{chunk}
</chunk>

Write 1-3 sentences situating this chunk in the document. Include:
- topic
- referenced entity/client/product
- relevant date if applicable

Don't repeat info from the chunk; only context.
"""

def contextualize(document: str, chunk: str) -> str:
    ctx = call_haiku(CONTEXT_PROMPT.format(document=document, chunk=chunk))
    return f"{ctx}\n\n---\n\n{chunk}"

Anthropic-measured impact: recall@20 rises from 5.7% to 1.9% failures (67% reduction). We see analogous improvement in our clients.

Step 4 — Dense + sparse embedding

Dense embedding captures semantics ("monthly fee" ≈ "periodic subscription"). Sparse (BM25) embedding captures exact match (IDs, proper names, codes).

from fastembed import TextEmbedding, SparseTextEmbedding

dense = TextEmbedding("BAAI/bge-small-en-v1.5")  # or 'intfloat/multilingual-e5-large'
sparse = SparseTextEmbedding("Qdrant/bm25")

dense_vec = next(dense.embed([chunk_text]))
sparse_vec = next(sparse.embed([chunk_text]))

For Spanish, we use intfloat/multilingual-e5-large (768 dims) or jinaai/jina-embeddings-v3 (1024 dims). BGE-m3 produces dense + sparse in a single pass.

Step 5 — Qdrant with hybrid search

Collection with two named vectors:

from qdrant_client import QdrantClient, models

client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)

client.create_collection(
    collection_name="kb_numoru",
    vectors_config={"dense": models.VectorParams(size=768, distance=models.Distance.COSINE)},
    sparse_vectors_config={"bm25": models.SparseVectorParams(modifier=models.Modifier.IDF)},
)

client.upsert(
    collection_name="kb_numoru",
    points=[
        models.PointStruct(
            id=uuid4().hex,
            vector={"dense": dense_vec, "bm25": sparse_vec.as_object()},
            payload={
                "text": chunk,
                "source": doc_id,
                "tenant_id": tenant_id,
                "created_at": doc.created_at.isoformat(),
                "category": doc.category,
            },
        )
    ],
)

Hybrid query with RRF fusion:

results = client.query_points(
    collection_name="kb_numoru",
    prefetch=[
        models.Prefetch(query=dense_vec, using="dense", limit=40),
        models.Prefetch(query=sparse_vec.as_object(), using="bm25", limit=40),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    query_filter=models.Filter(must=[
        models.FieldCondition(key="tenant_id", match=models.MatchValue(value=tenant_id))
    ]),
    limit=40,
)

Step 6 — Reranking with BGE-reranker v2-m3

The top-40 from hybrid search has decent precision but still includes noise. The reranker re-evaluates (query, chunk) pairs with a cross-encoder and reorders.

from FlagEmbedding import FlagLLMReranker

reranker = FlagLLMReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

def rerank(query: str, hits, top_k=6):
    pairs = [[query, h.payload["text"]] for h in hits]
    scores = reranker.compute_score(pairs, normalize=True)
    ranked = sorted(zip(hits, scores), key=lambda x: -x[1])
    return [h for h, _ in ranked[:top_k]]

Running the self-hosted reranker on CPU for 40 pairs takes ~250 ms. On a small GPU, <50 ms. Typical nDCG@5 improvement: +15-22 points.

Alternatives:

• Cohere Rerank v3 (SaaS) — excellent multilingual, $0.002 per rerank.
• Jina Reranker v2 — competitive open source.

Step 7 — RAPTOR for hierarchical documentation

When documents are long (manuals, regulations, books), flat retrieval fails on questions like "what's the general theme of chapter 3?". RAPTOR builds a tree of summaries:

1. Base chunks (tree leaves).
2. Cluster related chunks (by similarity).
3. Generate a summary per cluster (parent).
4. Repeat until you have a root.
5. Embed each level.

At query time you search every level simultaneously. Specific questions land at leaves; general questions land at summaries.

from raptor import RetrievalAugmentation

RA = RetrievalAugmentation(config=raptor_config)
RA.add_documents(document_texts)
answer = RA.answer_question(question=query)

For most cases (short docs, FAQs, emails) RAPTOR is overkill. We use it for legal and technical manuals.

Semantic cache with RedisVL

Before querying Qdrant, check Redis: did anyone make a "sufficiently similar" query in the last 24h?

from redisvl.extensions.llmcache import SemanticCache

cache = SemanticCache(
    name="rag_cache",
    redis_url=os.getenv("REDIS_URL"),
    distance_threshold=0.12,  # ≈ similarity 0.88
    ttl=86400,
)

def answer(query: str) -> str:
    if hit := cache.check(query):
        return hit[0]["response"]

    ctx = rag_retrieve(query)
    response = llm(ctx, query)
    cache.store(prompt=query, response=response)
    return response

Typical hit rate in support agents: 35-55%. Saves latency and tokens.

Evaluation with Ragas + DeepEval

RAG is non-deterministic; it needs systematic eval. Three indispensable metrics:

• Faithfulness — is the answer derivable from retrieved chunks?
• Answer Relevancy — does the answer address the query?
• Context Recall — do retrieved chunks cover the expected answer?

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision

dataset = load_golden_rag_v2()
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_recall, context_precision],
    llm=claude_sonnet,
)

Scores are written to Langfuse as evals on the dataset run. In CI we integrate with Promptfoo + DeepEval: a PR that drops context_recall more than 5 points is blocked.

Real metrics from a client case

Legal client, 8,000 documents, 120 golden queries.

Each layer contributes. The combination separates production from demo.

Cost at scale

Case: 50,000 indexed chunks, 100k queries/month.

Without cache, Sonnet alone, no reranker: $800-950. The stack cuts cost 45% while raising precision.

Business & commercial impact

Common anti-patterns

1. 500-token chunks because "someone said it's optimal". It depends on the domain. Try 256, 512, 1024 and measure.
2. Embed everything with text-embedding-ada-002 (v1). Obsolete; text-embedding-3-small is 5× better and cheaper.
3. Not filtering by tenant_id. Critical in multi-tenant. A missing filter = data leak.
4. Forgetting to update the index when a doc changes. Without TTL or upsert by doc_id, the index drifts out of sync.
5. Passing the LLM the top-40 without rerank. Noise degrades the answer more than it helps.
6. Not measuring. Without Ragas or DeepEval in CI, regressions slip by silently.

When NOT to do RAG

• If relevant data fits in the prompt (<100k tokens) and long-context latency is acceptable.
• If the domain changes faster than re-indexing (seconds-minutes).
• If the cost of a fictitious answer is catastrophic — use deterministic search + LLM as formatter only.

FAQ

What chunk size should I use?Start with 512 tokens + 50 overlap. Measure. Adjust. For code: 256 often wins; for regulation: 768-1024.

Local or SaaS embedding? SaaS while volume is <500k embeddings/month. Beyond that, BGE-m3 local on CPU is 10× cheaper.

Which reranker model?BGE-reranker v2-m3 is the best multilingual OSS today. Cohere Rerank v3 if you prefer SaaS.

RAG vs long context?Both work. RAG controls attribution and cost better; long context is simpler. Use long context for short conversations with a single document; RAG for large KBs and multi-tenant.

How do I handle documents that update?Upsert by doc_id as the key. When a new version arrives, re-chunk, re-embed and replace all chunks for that doc. Use schedule_date in payload to mark freshness.

Next steps

The full pipeline is at github.com/numoru-ia/rag-production-stack with a Docker Compose that brings up Qdrant + Redis + Langfuse + FastAPI + indexing scripts. The next piece covers orchestrating three agents on top of this RAG for a concrete vertical case.