Agent Memory vs RAG vs Long Context: Which One You Need (With Costs)

Agent memory, RAG and long context are not three answers to the same question. Long context makes the model read everything on every request. RAG (retrieval-augmented generation) searches a large shared corpus and sends only the relevant chunks. Agent memory stores facts about a user or task and recalls them in later sessions.

They also cost very different amounts. In the worked example below, answering a question over a 250K-token knowledge base costs about $0.51 per query with raw long context, $0.084 with prompt caching, and $0.012 with RAG. Below roughly 20K tokens of knowledge, you don’t need RAG at all.

The three approaches at a glance

Long context: just send everything

Context windows are now big enough that “just put it all in the prompt” is a real option. Claude 4.6 and later models have a 1M-token window, billed at the same per-token rate for a 900K-token request as for a 9K-token one (Anthropic pricing, October 2026). Not every provider prices this way, so check your provider’s pricing page for a higher rate above a certain prompt size.

Two things work against it:

• Cost scales with every token, on every request. A 250K-token prompt is paid for 60,000 times if you get 60,000 questions.
• Accuracy drops as input grows. Chroma’s Context Rot study (July 2025) tested 18 models and found performance degrades as input length increases, even on simple tasks, and depends on where the relevant text sits in the prompt.

Prompt caching softens the first problem a lot. The second it doesn’t touch.

RAG: search first, then answer

RAG splits documents into chunks, embeds them as vectors, and at query time sends only the closest matches to the model. It scales to corpora far bigger than any context window and keeps each request small.

What it gives up is completeness. The model can only reason over what retrieval returned. Questions that need the whole picture, like “which of our 40 runbooks contradict each other?”, are where RAG quietly fails.

Agent memory: remember the user, not the library

Agent memory is about continuity: preferences, decisions and facts that came up in earlier conversations. A memory layer typically runs an extra model call after each turn to extract facts worth keeping, stores them, and recalls the relevant ones next time.

2025 and 2026 brought a wave of dedicated frameworks, including Mem0, Zep (and its Graphiti graph engine), Letta and LangMem. The model providers ship their own versions too. Anthropic’s memory tool lets Claude create and edit files in a /memories directory that your code stores. Its Managed Agents platform mounts versioned memory stores as a directory, currently in beta.

Benchmarks exist (LoCoMo and LongMemEval are the common ones), but treat vendor numbers as directional. The Mem0 paper reports 91% lower p95 latency and over 90% token savings versus a full-context baseline on LoCoMo. Different vendors publish conflicting results for the same benchmarks, so test on your own data.

What each approach costs per query

Prices change and every workload is different, so here is the cost model as code you can rerun with your own numbers. It uses Claude Sonnet 5.5 list prices from October 2026:

• input: $2 per million tokens
• 5-minute cache write: $2.50
• cache read: $0.20
• output: $10

Memory extraction runs on Claude Haiku 4.5 at $1 input and $5 output.

"""Cost per query for long context, RAG, agent memory and curated docs.

Prices: Claude API list prices, USD per million tokens, October 2026.
Check https://platform.claude.com/docs/en/about-claude/pricing before reusing.
"""

PRICES = {
    "sonnet-5.5": {"input": 2.00, "cache_write": 2.50, "cache_read": 0.20, "output": 10.00},
    "haiku-4.5": {"input": 1.00, "cache_write": 1.25, "cache_read": 0.10, "output": 5.00},
}
EMBEDDING_PER_MTOK = 0.02  # a small hosted embedding model


def call_cost(model, fresh=0, cached=0, cache_hit_rate=0.0, output=0):
    """One LLM call. `cached` tokens are a stable prefix that is either read
    from the prompt cache (hit) or written to it (miss)."""
    p = PRICES[model]
    prefix = cached * (cache_hit_rate * p["cache_read"] + (1 - cache_hit_rate) * p["cache_write"])
    return (fresh * p["input"] + prefix + output * p["output"]) / 1_000_000


def embed_cost(tokens):
    return tokens * EMBEDDING_PER_MTOK / 1_000_000


# Scenario 1: Q&A over a 250K-token internal knowledge base
KB_TOKENS = 250_000
PROMPT = 600     # system prompt + question
ANSWER = 400     # output tokens
HIT_RATE = 0.95  # steady traffic keeps the 5-minute cache warm

scenario_1 = {
    "Long context, no caching": call_cost("sonnet-5.5", fresh=KB_TOKENS + PROMPT, output=ANSWER),
    "Long context, prompt caching": call_cost(
        "sonnet-5.5", fresh=PROMPT, cached=KB_TOKENS, cache_hit_rate=HIT_RATE, output=ANSWER),
    "RAG (8 chunks x 400 tokens)": call_cost("sonnet-5.5", fresh=8 * 400 + PROMPT, output=ANSWER)
    + embed_cost(30),
    "Curated docs (12K tokens, cached)": call_cost(
        "sonnet-5.5", fresh=PROMPT, cached=12_000, cache_hit_rate=HIT_RATE, output=ANSWER),
}

# Scenario 2: an assistant that remembers one user across 40 past sessions
HISTORY_TOKENS = 150_000  # everything the user ever said, plus replies
RECENT = 6_000            # the current conversation so far
TURNS_PER_SESSION = 8     # the first turn writes the cache, the rest read it
session_hit_rate = (TURNS_PER_SESSION - 1) / TURNS_PER_SESSION

scenario_2 = {
    "Replay full history (cached in session)": call_cost(
        "sonnet-5.5", fresh=RECENT + PROMPT, cached=HISTORY_TOKENS,
        cache_hit_rate=session_hit_rate, output=ANSWER),
    "Agent memory (2K memories + extraction)": call_cost(
        "sonnet-5.5", fresh=2_000 + RECENT + PROMPT, output=ANSWER)
    # write path: a cheaper model extracts facts from every turn
    + call_cost("haiku-4.5", fresh=3_000, output=200)
    + embed_cost(2 * 200),
    "Memory file (5K profile, edited by tool)": call_cost(
        "sonnet-5.5", fresh=5_000 + RECENT + PROMPT, output=ANSWER + 150),
}


def report(title, rows, per_month):
    print(title)
    for name, cost in rows.items():
        print(f"  {name:<42} ${cost:>8.4f}/query   ${cost * per_month:>9,.0f}/month")
    print()


report("Scenario 1: knowledge-base Q&A (60,000 queries/month)", scenario_1, 60_000)
report("Scenario 2: personal assistant (100,000 turns/month)", scenario_2, 100_000)

# Below this corpus size, caching the whole corpus costs less per query than RAG.
p = PRICES["sonnet-5.5"]
per_cached_token = (HIT_RATE * p["cache_read"] + (1 - HIT_RATE) * p["cache_write"]) / 1_000_000
same_for_both = call_cost("sonnet-5.5", fresh=PROMPT, output=ANSWER)
rag = scenario_1["RAG (8 chunks x 400 tokens)"]
print(f"Break-even corpus size: {(rag - same_for_both) / per_cached_token:,.0f} tokens")

Running it with Python 3.13 prints:

Scenario 1: knowledge-base Q&A (60,000 queries/month)
  Long context, no caching                   $  0.5052/query   $   30,312/month
  Long context, prompt caching               $  0.0840/query   $    5,037/month
  RAG (8 chunks x 400 tokens)                $  0.0116/query   $      696/month
  Curated docs (12K tokens, cached)          $  0.0090/query   $      539/month

Scenario 2: personal assistant (100,000 turns/month)
  Replay full history (cached in session)    $  0.0903/query   $    9,032/month
  Agent memory (2K memories + extraction)    $  0.0252/query   $    2,521/month
  Memory file (5K profile, edited by tool)   $  0.0287/query   $    2,870/month

Break-even corpus size: 20,319 tokens

What the numbers say

• Prompt caching is the first fix for long context, not the last. It cuts the cost about six-fold, from $0.505 to $0.084 per query, but RAG is still about seven times cheaper again.
• Small knowledge doesn’t need RAG. Below about 20K tokens, a cached prompt containing everything costs less per query than retrieval, with no vector database to run and no chunks to get wrong.
• Replaying full chat history is the expensive way to remember users. Even with in-session caching, it costs about 3.6 times more than a memory layer in scenario 2, and it gets worse as history grows.
• Memory’s hidden cost is the write path. The extraction call on every turn is about a fifth of the per-turn cost. A memory file that the model edits only when something changes avoids that call but pays in output tokens.
• The model leaves out infrastructure. Vector database hosting, re-embedding when documents change, and the engineering time to tune retrieval all land on the RAG and memory side of the ledger.

The counter-take: maybe you need documentation, not memory

The most useful argument in this debate isn’t about any of the three. In “Agents don’t need memory, they need documentation” (3 October 2026), Kevin Liao argues that agent memory plugins are mostly RAG over old chats. He proposes instead that agents read a folder of plain Markdown docs before a task and update them afterwards. The Hacker News thread drew several hundred points and a long argument.

The pushback in the thread is as instructive as the post:

• Stale docs can be worse than none. An old decision left next to a new one confuses the agent more than silence would.
• Agents are bad editors. Documentation written by an agent tends to pick up speculation and one-off details that bloat it over time.
• Code only shows the “what”. The “why”, the constraints and team preferences, is exactly what docs capture and code can’t.

Here is how that maps onto the comparison. Curated docs are the “long context, but small” option, and in scenario 1 they were the cheapest of all. They work when the knowledge is small, slow-changing and owned by people who review it. They stop working when nobody maintains them, or when the knowledge is personal to each user. That is the job agent memory was built for.

The platforms seem to be converging here too. Both of Anthropic’s memory features store memories as plain files that people can read, edit and roll back, rather than as opaque vectors.

How to choose

Work through these in order:

1. Do you need to reason over whole documents at once? Contract review, comparing every runbook, a full codebase audit: use long context with prompt caching.
2. Does the shared knowledge fit in about 20K tokens? Use curated docs in the system prompt, cached. Review them like code.
3. Is the knowledge about a specific user or session? Use agent memory, or a memory file the model edits.
4. Otherwise, a large, shared and changing corpus is what RAG is for.

Most production systems combine two of these. A common shape is a cached system prompt with curated docs, RAG over the big corpus, and a small memory layer for user preferences. If your RAG index has to stay fresh as documents change, stream the changes through a queue and re-embed them in a consumer group (see how Kafka consumer groups divide that work).

Common mistakes

• Building RAG for 15K tokens of docs. You add a vector store, a chunker and retrieval failures to save a few cents per thousand queries.
• Stuffing long context without caching. In the example above, that’s the difference between $30,312 and $5,037 a month.
• Memory with no expiry. Users change jobs, preferences and stacks. A memory layer needs a way to update or retire facts, not just add them.
• Trusting vendor benchmarks. Run a small evaluation on your own conversations before choosing a memory framework.
• Storing secrets in memory. Anything written to memory is replayed into future prompts. Never store API keys, tokens or passwords there, and check privacy rules before storing personal data.
• Comparing cost per request instead of cost per resolved task. A cheaper request that needs three follow-ups isn’t cheaper.

FAQ

Is RAG dead now that context windows are 1M tokens?

No. Bigger windows remove the hard size limit, but not the per-token cost or the accuracy loss as input grows. For a large corpus queried often, RAG is still several times cheaper per query, even against cached long context.

Does prompt caching make long context as cheap as RAG?

Not for large corpora. Caching made long context about six times cheaper in the example above, but RAG was still about seven times cheaper than that. Caching only beats RAG on cost when the whole knowledge base is small, roughly under 20K tokens at these prices.

Is agent memory just RAG over old conversations?

Often, yes. Many memory layers extract facts, embed them and retrieve the closest ones. The difference is what gets stored: distilled, updatable facts about a user rather than raw document chunks. Good memory systems also handle updates and contradictions, which plain RAG doesn’t.

Do I need a vector database for agent memory?

Not necessarily. For small, per-user memories, a file or a database row the model reads in full is simpler and easy to audit. Vector search earns its place once each user’s memory is too large to load every turn.

How do I keep long context costs down?

Cache the stable part of the prompt and keep it at the start, byte-identical across requests. Put anything that changes per request after it. Use batch processing for work that doesn’t need an instant answer; on the Claude API it costs half, and the discount stacks with caching.

Bottom line

Start with the cheapest thing that works:

1. Curated docs in a cached prompt.
2. Add RAG when the shared knowledge outgrows about 20K tokens.
3. Add memory when the product needs to remember individual users.

Long context is the right tool when the task is to read everything at once, not a substitute for deciding what the model should read. Rerun the cost model above with your own corpus size, traffic and model prices before you build.