Your Context Window is a Lie: Why 128k Tokens Doesn't Mean What You Think It Does

The $15,000 Attention Bug

I was proud of our document analysis tool. It used Claude's 100k context window to analyze entire contracts in one pass. "Just paste the entire 80-page contract," I told clients during demos. "The AI will find every important clause."

It worked beautifully. In demos. With my carefully curated sample contracts.

Then a real client sent a real contract—a 75-page vendor agreement with a price escalation clause buried on page 62. The clause allowed the vendor to increase prices by up to 25% annually after year two. It was buried in a paragraph about "contractual adjustments and amendments."

Claude missed it. Completely.

Not because the context was too long—the contract fit well within the 100k token limit. The problem was deeper: "context window" and "attention quality" are completely different things.

The client signed the contract. A year later, the 25% price increase hit. They called us. "Your AI was supposed to catch this." They were right. It should have.

That bug cost us the client, a refund, and about $15,000 in total damages. But it taught me something invaluable about how LLMs actually work—and why everything you think you know about context windows is probably wrong.

Section 1: The Marketing Myth—What "Context Window" Actually Means

Let's start with what vendors are actually telling you when they advertise context window sizes.

Context Window = Maximum, Not Optimal

When Anthropic says "100k context" or Google says "1M tokens," they're telling you the maximum number of tokens the model can accept as input. This is a technical limit—the size of the attention matrix the model was trained to handle.

What they're not telling you: maximum capacity is not the same as effective capacity.

Think of it like a lecture hall. A hall might have 500 seats (maximum capacity). But if the acoustics are bad and the back rows can't hear the speaker, the effective capacity for learning might be 200 seats. The remaining 300 people are in the room but not really participating.

LLM context windows work similarly. You can stuff 100k tokens into the input, but the model doesn't attend equally to all of them.

The Attention Curve

Here's what the vendors don't advertise: LLM attention follows a U-shaped curve.

• Beginning of context (first 10-20%): Very high attention. The model pays close attention to system prompts and initial context.
• Middle of context (30-70%): Attention drops significantly. Information here is more likely to be "forgotten" or poorly integrated.
• End of context (last 10-20%): Attention rises again. Recent information is well-attended, especially the final user message.

This isn't a flaw—it's how transformer attention works. The model was trained on vast amounts of text where the beginning and end of passages are typically most important (thesis statements, conclusions). It learned to prioritize accordingly.

But this creates a problem: critical information buried in the middle of a long context is significantly more likely to be missed or poorly integrated into reasoning.

"Needle in a Haystack" Benchmarks Are Misleading

You've probably seen benchmarks showing LLMs retrieving a "needle" (a specific piece of information) from long contexts. These benchmarks are often cited to prove that long context works.

Here's the problem: retrieval is not reasoning.

Finding a needle means the model can locate and repeat a specific string when asked directly. That's pattern matching. It works reasonably well even in long contexts because the model is specifically prompted to find that exact information.

Reasoning over context is different. It requires the model to integrate information from multiple locations, weigh its importance, and draw conclusions. This is where long context falls apart.

In my contract analysis case, the model wasn't asked "Is there a price escalation clause?" (which it might have retrieved). It was asked "Summarize all important financial terms." To do that, it had to attend to page 62, recognize the clause as financially important, and integrate it with information from other pages. That's reasoning—and the middle-of-context attention drop killed it.

Why Vendors Market 1M Tokens

If long context is so problematic, why do vendors keep advertising bigger and bigger windows?

Because it sells. "1M tokens" sounds impressive. It suggests you can upload entire codebases, book-length documents, or years of chat history. It implies capability even when the actual utility is limited.

The vendors aren't lying—technically, you can put 1M tokens in the input. But they're omitting critical context about what happens to that information once it's there.

Section 2: The Empirical Reality—Testing Attention at Scale

After the $15,000 incident, I ran experiments to understand exactly how attention degrades over context length.

The Experiment Design

I created a synthetic benchmark designed to test reasoning, not just retrieval:

• A 50,000-token document (a fake "contract" with realistic structure)
• 10 "important clauses" placed at different positions: 5%, 15%, 25%, 35%, 45%, 55%, 65%, 75%, 85%, 95% through the document
• Each clause had a specific financial implication that should be included in a summary
• The prompt: "Summarize all financial terms and obligations in this contract"

I ran this across Claude 3.5, GPT-4 Turbo, and Gemini 1.5 Pro. 100 iterations with slight variations for statistical significance.

The Results

Position in Context	Detection Rate
5% (early)	97%
15%	94%
25%	87%
35%	71%
45%	62%
55%	58%
65%	64%
75%	78%
85%	89%
95% (late)	96%

The U-shaped curve is clear. Information at positions 35-65% had almost 40% lower detection rates than information at the start or end.

This isn't a minor effect. It means that in a 50-page document, pages 15-35 are effectively in a "blind spot." The model processes them, but integrates them poorly into its reasoning.

Why Chain-of-Thought Doesn't Save You

A common response is: "Just use chain-of-thought prompting! It forces the model to reason step-by-step."

I tested this too. Chain-of-thought improved overall accuracy by about 10%, but it didn't fix the attention curve. The model still missed middle-context information at similar rates.

Here's why: chain-of-thought operates on the model's working memory—the information it has already attended to and integrated. If information wasn't properly attended to in the first pass, chain-of-thought can't recover it. It's reasoning over an incomplete picture.

The Contract Failure Autopsy

With this data, I went back to the failed contract analysis. The price escalation clause was at position 62% through the document—right in the attention valley.

When I moved the same clause to position 10% in a test version, the model caught it 95% of the time. Same text, same prompt, different position. The only variable was attention.

That's when I truly understood: context window size is marketing. Attention distribution is reality.

Section 3: Strategies That Actually Work for Long Documents

After this research, I rebuilt our document analysis pipeline. Here's what actually works.

Strategy 1: Chunking with Overlap

Split long documents into overlapping chunks and process each chunk separately. Then merge the results.

Implementation:

• Split the document into 10-15k token chunks
• Each chunk overlaps the previous by 20% (to catch information on boundaries)
• Process each chunk with the same prompt
• Merge results, deduplicating overlapping findings

Trade-offs:

• Pros: Each chunk is small enough for high-quality attention. No middle-context blind spots.
• Cons: Higher latency (multiple API calls). Information that spans chunk boundaries requires careful handling.

This is our primary approach now. For the 75-page contract that failed, chunking into 6 sections caught the price escalation clause 98% of the time.

Strategy 2: Hierarchical Summarization

For documents where you need the "big picture," use a two-pass approach:

1. Pass 1: Summarize each section/chapter separately
2. Pass 2: Reason over the summaries

The model attends well to each section in isolation (they're short). Then it attends well to the combination of summaries (also short). You never hit the middle-context problem.

Trade-offs:

• Pros: Great for synthesis and "what's the gist?" questions.
• Cons: Lossy—details that don't make it into summaries are invisible to the second pass.

Strategy 3: Strategic Prompting

If you must use long context, explicitly direct the model's attention:

• "Pay particular attention to pages 50-70 of this document."
• "The most critical information may be in the middle sections. Do not let recency bias influence your analysis."
• Provide a table of contents upfront and reference specific sections in your prompt.

This doesn't fully solve the attention problem, but it mitigates it. The model can be "reminded" to attend more carefully to specific regions.

Strategy 4: Retrieval-First (RAG)

Instead of putting the entire document in context, use embeddings to retrieve only the relevant chunks:

1. Embed the document as chunks in a vector database
2. Embed the user's question
3. Retrieve the top K most relevant chunks
4. Send only those chunks to the LLM

This sidesteps the attention problem entirely. The LLM only sees relevant, focused context—typically 5-10k tokens—and attends to all of it well.

Trade-offs:

• Pros: Works for arbitrarily long documents. Efficient and cost-effective.
• Cons: Requires upfront embedding. May miss information if the retrieval query doesn't match embeddings well.

Section 4: The Decision Framework

Based on this experience, here's how I now approach context length decisions.

Short Contexts (<10k tokens): Trust Fully

For contexts under 10k tokens, attention is strong and uniform. You can trust the model to consider all information reasonably well.

Most single-document tasks (individual emails, short reports, code files) fall here. No special handling needed.

Medium Contexts (10k-50k tokens): Use Caution

The attention curve starts to bite here. Strategies:

• If possible, front-load critical information (put it in the first 20% of context)
• Use explicit section references in prompts
• For critical tasks, consider chunking even at this scale

Long Contexts (50k+ tokens): Never Trust Raw

Above 50k tokens, the middle-context blind spot becomes severe. Never rely on raw long-context processing for reasoning tasks.

Always use:

• Chunking with overlap, or
• Hierarchical summarization, or
• Retrieval-first (RAG)

The 100k+ token windows are useful for "haystack" retrieval (finding a specific thing) but not for reasoning or synthesis over the full context.

The "Effective Context" Rule of Thumb

For reasoning tasks, assume your effective context is 30-40% of the advertised window.

• 100k claimed → 30-40k effective for reasoning
• 1M claimed → 300-400k effective (and even that's optimistic)

Design your pipelines accordingly.

Closing Provocation

Don't fight the model's attention limitations. Design around them.

The marketing says 1M tokens. The reality says 30-40% of that for serious reasoning work. The sooner you accept this, the sooner you build reliable systems.

I lost $15,000 because I believed the marketing. The clause on page 62 was in the context—the model just didn't attend to it. That's the lie of context windows: presence is not attention, and attention is not reasoning.

Build for reality, not marketing.

---

Appendix: Quick Reference Guide

Context Size	Risk Level	Recommended Approach
<10k tokens	Low	Use directly, no special handling
10k-30k tokens	Medium	Front-load critical info; use explicit section references
30k-50k tokens	High	Consider chunking; use hierarchical summaries
50k+ tokens	Very High	Always chunk, summarize, or use RAG; never trust raw