You gave your LLM a 200k context window and the answers got worse. This is not a bug in the model. It is a well-documented property of how attention mechanisms work at scale, and it has a name: the lost in the middle problem.
What the Research Shows
A 2023 paper by Liu et al. demonstrated that language models consistently perform worse when relevant information is placed in the middle of a long context window compared to the beginning or the end. The effect is significant and reproducible across model families including GPT-4, Claude, and open-source alternatives.
The mechanism is attention distribution. Transformers allocate attention weights across all tokens in the context, but the distribution is not uniform. Tokens at the start of a sequence and tokens near the task instruction at the end receive disproportionately high attention. Everything in between competes for what remains.
In practice this means a 50-document RAG injection where your most relevant chunk lands at position 25 will often produce worse results than a 3-document injection where that same chunk is first.
How This Breaks Production RAG Systems
The standard RAG implementation retrieves the top-K results by similarity score, appends them to a prompt in retrieval order, then appends the user query. This puts the highest-scoring chunk first and the query last — which is actually reasonable. The problem is K.
Most teams set K to 10 or higher in the belief that more context is better. At K=10 with chunks of 500 tokens each, you are injecting 5,000 tokens of context before the query. Chunks 4 through 8 are landing squarely in the middle of the attention distribution where the model is least focused.
The result is a model that tends to answer from chunks 1-2 and occasionally chunk 10, while ignoring chunks 3-9 entirely — even if chunk 6 contains the most relevant information.
Rules That Fix It
Put your best chunk first. After retrieval, rerank your results by a combination of similarity score and estimated relevance to the specific query. Place the highest-confidence chunk at position 0, not after a preamble.
Put your task instruction last. The model reads the full context and then encounters the instruction. This sequencing means the instruction lands with full context available. Burying the instruction in the middle of injected documents causes it to compete with surrounding context for attention.
Cap retrieval to top-3. Studies on retrieval tasks consistently show that performance peaks at roughly 20 to 40 percent of the available context window. For a 16k context model, that is roughly 3,000 to 6,000 tokens of injected content — about 3 to 6 chunks at standard chunk sizes. More than that and you are adding noise faster than you are adding signal.
Filter before you inject. Apply a relevance threshold to retrieved chunks before including them. A chunk with a cosine similarity of 0.62 is probably noise. Set a minimum threshold — 0.75 is a reasonable starting point — and exclude chunks below it regardless of K.
The Mental Model
Context is not storage. It is working memory.
A human expert asked a specific question does not perform better when handed a larger stack of reference material. They perform better when handed the right page. The same principle applies to language models. The discipline of selecting and positioning context is at least as important as the quality of the base model.
What This Looks Like in the ry-ops Stack
The narrative tracking system in fabric/social uses Qdrant for semantic search over previously used content angles. At retrieval time, it pulls the top-3 results by cosine similarity, filters anything below 0.78, and injects them at the beginning of the generation prompt before the source content and after the system instructions.
The task instruction — generate a carousel that does not repeat previously used angles — lands at the end of the prompt. This is not accidental. It is the direct application of the position-matters principle.
The context window for this call is typically 15 to 20 percent utilized. That is intentional.
Summary
If you are building RAG systems and seeing inconsistent answer quality despite high retrieval scores, check three things. First, where are your retrieved chunks positioned relative to the task instruction. Second, what is your K value and whether you have a relevance threshold. Third, whether you are reranking after retrieval or serving results in raw similarity order.
Context is not free. Every token you inject is competing for the model’s attention. Spend it deliberately.
Frequently Asked Questions
What is the lost in the middle problem in LLMs?
The lost in the middle problem is a documented behavior where language models give less weight to information positioned in the middle of a long context window. Transformers allocate disproportionate attention to tokens at the start and end of a sequence, causing middle content to be underweighted during generation. It was formally demonstrated in a 2023 paper by Liu et al. across GPT-4, Claude, and open-source models.
How does the lost in the middle problem affect RAG systems?
In a standard RAG pipeline that injects the top-K retrieved chunks before the user query, chunks ranked 4 through K-1 land in the middle of the attention distribution where the model is least focused. This means a highly relevant chunk at position 6 of 10 may be effectively ignored even if it scores highest on semantic similarity. Reducing K and reranking to ensure the most relevant chunk appears first significantly mitigates this.
What is the optimal value for K in a RAG retrieval pipeline?
There is no universal optimal K, but research and production experience suggest that smaller values — often 3 to 5 chunks — outperform larger values like 10 or 20 when chunk quality is high. The key principle is that precision matters more than recall: a smaller set of highly relevant chunks placed at the beginning of the context outperforms a larger set where relevant content is buried in the middle.
How can you fix the lost in the middle problem in production?
Three practical mitigations are: first, rerank retrieved chunks so the highest-relevance document appears first in the injected context; second, reduce K to limit how much content lands in the attention-sparse middle region; third, place the user query immediately after the most critical chunk rather than at the very end of a long context block. Combining retrieval similarity scores with a cross-encoder reranker is the most reliable approach.
