Why We Deleted Our Vector Database. RAG is Dead, Long Live 'Context Caching'.

We had a RAG (Retrieval Augmented Generation) pipeline. It was beautiful. We chunked 50,000 internal documents. We embedded them into Pinecone. We built a semantic search layer with re-ranking.

It worked—sort of. Query latency was 2 seconds. Relevance was mediocre (we spent months tweaking chunk sizes and overlap). And then I saw the bill: $8,000/month for hosting vectors.

I asked the team: "How many times do we actually query this thing?"

The answer: 50 queries per day.

We were paying $5.33 per query. For a glorified Ctrl+F.

I deleted the Vector Database that weekend. We switched to "Context Caching" with Gemini 1.5 Pro's 2 Million token context window.

Latency dropped to 200ms. Cost dropped to $50/month. Relevance went up because the model sees the entire document, not a 500-token snippet.

Here is why RAG is already obsolete for most enterprise use cases, and what you should build instead.

Section 1: The "Chunking" Nightmare

RAG requires you to "chunk" your documents. The standard advice is 500-1000 tokens per chunk with some overlap.

This sounds reasonable until you actually do it.

The Context Destruction Problem:

Documents are not bags of independent sentences. They are narratives. A paragraph references the previous paragraph. A section builds on the section before it.

When you chunk, you destroy these references. The model receives a 500-token snippet that says "As mentioned above..." but the "above" is in a different chunk that wasn't retrieved.

The result? Hallucinations. The model guesses what "above" means.

The Semantic Search Lie:

Semantic search (embedding similarity) is supposed to find the "most relevant" chunks. In practice, it finds the chunks with the most similar words, not the most relevant meaning.

If you ask "What is our refund policy?", semantic search might return a chunk about "product returns" (similar words!) instead of the actual "Refund Policy" section (which uses different vocabulary like "reimbursement" and "credit").

You end up building complex "Re-Ranker" pipelines (using another LLM to re-rank the retrieved chunks). This adds latency, cost, and complexity. You are building a Rube Goldberg machine to fix a fundamentally broken approach.

The Hyperparameter Hell:

Chunk size? Chunk overlap? Which embedding model? Which similarity metric (cosine, dot product, Euclidean)? Top-K retrieval? Re-ranker threshold?

Each of these is a hyperparameter you have to tune. For each document type. Forever.

We spent 3 months tuning these knobs. Every time we added new documents, the old settings broke.

Section 2: The Rise of the "Mega Context Window"

While we were fighting RAG, the foundation model labs were solving the problem at the root.

The Numbers:

• Gemini 1.5 Pro: 2,000,000 tokens (1,500+ pages of text).
• Claude 3: 200,000 tokens.
• GPT-4 Turbo: 128,000 tokens.

For most enterprise use cases (company knowledge bases, legal contracts, HR policies, support ticket history), the entire corpus fits in the context window.

The Math:

Our 50,000 documents were mostly short (meeting notes, policies, FAQs). Total token count: ~1.2 Million tokens.

That fits in Gemini 1.5 Pro's window. No chunking. No embedding. No vector database.

You just... paste the documents into the prompt.

The Relevance Improvement:

When the model sees the entire corpus, it doesn't have to guess context. It can follow references. It can synthesize information from multiple sections.

Our "Answer Accuracy" (measured by human eval) went from 72% (with RAG) to 91% (with full context). A 19-point jump by doing less engineering.

Section 3: "Context Caching" (The New Paradigm)

Putting 1.2M tokens in every prompt sounds expensive. It would be — if you paid for it every time.

Enter Context Caching.

How It Works (Gemini Example):

1. You upload your entire document corpus once.
2. Gemini "caches" it. The cache has a TTL (Time to Live) — typically 1 hour to 24 hours.
3. Subsequent queries reference the cached context. You pay a fraction of the cost (just the query tokens, not the corpus tokens).

The Cost Breakdown:

• Initial Cache Creation: ~$10 (one-time per refresh).
• Per Query: ~$0.001 (just the output tokens).
• Daily Cost (50 queries): ~$0.05.
• Monthly Cost: ~$50 (including cache refreshes).

Compare this to $8,000/month for our Pinecone + Re-Ranker + Embedding pipeline.

The Latency Win:

RAG requires multiple round-trips: Query -> Embedding API -> Vector DB -> Retrieve -> Re-Rank -> LLM.

Context Caching is one round-trip: Query -> LLM (with cached context).

Our P95 latency dropped from 2.1 seconds to 180 milliseconds. Users noticed. NPS went up.

Section 4: When RAG Still Makes Sense (And When It Doesn't)

I am not saying RAG is useless. It is a tool. But it is often the wrong tool.

RAG Makes Sense When:

• Corpus is Massive (100M+ tokens): If you are indexing all of Wikipedia or a legal database with millions of documents, no context window is big enough.
• Data Changes Hourly: If your documents are live (e.g., stock prices, news feeds), caching is expensive to refresh constantly. RAG's incremental indexing is better.
• Latency Doesn't Matter: For batch processing (overnight report generation), the extra latency of RAG is irrelevant.

RAG is Overkill When:

• Corpus is < 2M tokens: Most internal knowledge bases (HR policies, product docs, meeting notes) fall into this category.
• Corpus is Relatively Static: If documents change weekly or monthly, cache refreshes are cheap.
• Latency Matters: If users are waiting for an answer (chatbots, search), every millisecond counts.

The Heuristic:

Before building RAG, ask: "Does my entire corpus fit in a 2M token window?"

If yes, skip RAG. Use Context Caching. Ship in a week instead of a quarter.

Conclusion

The AI infrastructure industry wants you to build complex pipelines. They sell Vector DBs, Embedding APIs, and Orchestration Frameworks.

But the brute-force approach — just paste everything into the context — is often the right one.

Stop over-engineering. Delete your Vector DB. Embrace the Mega Context Window.