Context Engineering: Context Windows, Caching & Agent Memory

Q: Compaction vs context editing — what's the difference?

Both keep the transcript from overflowing, but compaction summarizes earlier turns into a condensed form (preserving gist, losing exact detail), while context editing removes stale content outright (old tool results, completed thinking). Use compaction when you need the substance of old turns; use editing when they're just dead weight. Many long-running agents use both.

Q: How is memory different from the context window?

The context window is what the model sees in a single call — ephemeral, rebuilt each request. Memory persists across sessions: a file or store the agent reads and writes, surviving between conversations. You store durable facts in memory and load only the relevant ones into context when needed, rather than keeping all history in-window forever.

Context Engineering

Context engineering is the discipline of deciding what information goes into the model's context window, when, and in what form — and doing it so the model performs well without runaway cost. Where prompt engineering asks "what should I say to the model?", context engineering asks the broader question: "what is the complete set of tokens the model sees on each call, and how do I curate it?"

It has become the central skill of building serious LLM applications. A chatbot with one message is trivial; an agent that runs for an hour, calls dozens of tools, and reads large documents is an exercise in context management. Get it right and the system is fast, cheap, and accurate. Get it wrong and it's slow, expensive, and confused — drowning in stale tool results or hitting the window limit mid-task.

TL;DR

The context window is the model's working memory — everything it can "see" in one call: system prompt, tools, history, retrieved data, tool results.
More context isn't always better. Irrelevant or stale content dilutes attention and raises cost; curate ruthlessly.
Prompt caching is the biggest cost lever — cache the stable prefix so repeated calls pay a fraction for it.
Long conversations need compaction (summarize old turns) or context editing (clear stale tool results) to stay under the window and keep cost sane.
Memory persists information across sessions (a file or store the agent reads/writes) — distinct from the in-window context of a single run.
Retrieval (RAG) is context engineering too: pulling only the relevant chunks into context instead of stuffing everything.

Quick Example

The highest-leverage move is caching the stable prefix so repeated requests don't reprocess it. Mark the frozen part of the prompt as cacheable:

The stable instructions are processed once and served from cache thereafter; only the varying question pays full price.

Core Concepts

The Context Window Is Working Memory

The context window is the fixed budget of tokens the model can attend to in a single call. Everything competes for it: the system prompt, tool definitions, conversation history, retrieved documents, images, and accumulated tool results. When the window fills, something must give — you truncate, summarize, or the request fails.

Two failure modes matter. Overflow: you exceed the window and the call errors or drops content. Dilution: you fit everything, but the signal is buried in noise — irrelevant history and stale tool output pull the model's attention away from what matters, degrading answer quality even below the limit. Context engineering fights both.

The Order That Everything Is Rendered

Models process context in a fixed order — roughly tools, then system prompt, then the message history. This ordering is what makes caching work: stable content goes first (cacheable prefix), volatile content goes last (a timestamp, the current question). Understanding the order lets you place things so the expensive stable part is reused and the cheap varying part sits at the end.

The Token Economics of Agents

An agent's transcript grows with every step. Each tool result stays in context by default, so by turn 50 you're reprocessing turns 1–49 on every call. Left unmanaged, cost grows quadratically with conversation length. This is why context engineering — not the framework you pick — is the thing that determines whether a long-running agent is affordable.

The Toolkit

Prompt Caching

Caching is a prefix match: the provider hashes the rendered prompt up to a marker and, on a later request with the identical prefix, serves those tokens from cache at a fraction of the cost (often ~10%). This is the single biggest cost optimization for repeated context.

The one rule that governs everything: any byte change anywhere in the prefix invalidates the cache from that point on. So keep the stable content frozen and byte-identical, put volatile content (timestamps, IDs, the current question) after the last cache marker, and serialize any JSON deterministically. A datetime.now() in your system prompt silently defeats caching entirely.

Compaction

When a conversation approaches the window limit, compaction summarizes earlier turns into a compact form, freeing space while preserving the gist. Some providers do this server-side automatically, returning a compaction block you pass back on the next request. It trades exact recall of old turns for the ability to keep going — the right trade for long sessions.

Context Editing

Context editing removes stale content rather than summarizing it — typically clearing old tool results or completed thinking blocks that are no longer relevant. Where compaction condenses, editing prunes. Both keep the transcript lean; use editing when old tool outputs are simply dead weight.

Memory

Memory is persistence across sessions, distinct from the in-window context of a single run. An agent given a memory tool reads and writes files (or a store) that survive between conversations — user preferences, learned facts, prior outcomes. It's how an assistant "remembers" you next week without keeping the entire history in context. The pattern: store durable facts in memory, load only the relevant ones into context when needed.

Retrieval

RAG is context engineering by another name: instead of stuffing an entire knowledge base into the window, retrieve only the chunks relevant to the current question and place those in context. It's how you give the model access to far more information than the window could hold, while keeping each call focused and cheap.

Best Practices

Curate, Don't Accumulate

Default to less context. Every token you add competes for attention and costs money. Include what the model needs for this step; drop what it doesn't. A focused 4K-token context often beats a bloated 40K one.

Freeze the Prefix for Caching

Put stable content (system prompt, tool definitions) first and never interpolate volatile values into it. Move the current date, session ID, and question to the end. Verify caching works by checking that cache-read tokens are non-zero across repeated requests.

Cap and Summarize Tool Output

Tool results are the fastest way to bloat context. Cap result sizes server-side, paginate large query results, and summarize verbose output before it enters context. A tool that can return a million rows eventually will.

Manage Long Runs Explicitly

For agents and long chats, enable compaction or context editing before you hit the window, not after. Set an iteration cap. Route sub-steps to cheaper models. These are architecture decisions no framework makes for you.

Separate Working Context from Long-Term Memory

Keep the in-window context small and current; push durable facts to a memory store and pull only what's relevant into context when needed. Conflating the two — keeping everything in-window forever — is how agents get slow and expensive.

Common Mistakes

Silently Invalidating the Cache

Letting Tool Results Pile Up

Stuffing Everything Instead of Retrieving

Dumping an entire knowledge base into the prompt "so the model has context" wastes tokens, dilutes attention, and may overflow the window. Retrieve the relevant slice (RAG) instead.

FAQ

Is a bigger context window always better?

No. A larger window lets you fit more, but stuffing it with marginally relevant content dilutes the model's attention and inflates cost — quality can drop even under the limit. Use the extra room when the task genuinely needs it (large documents, long agent runs), and keep context curated regardless of the window size.

How much can prompt caching save?

A lot — cached tokens typically cost around a tenth of full price, so an application with a large stable prefix (long system prompt, big tool set, shared document) reused across requests can cut input cost dramatically. The catch is the prefix must be byte-identical; a single varying value at the front defeats it. Verify with cache-read token counts.

Compaction vs context editing — what's the difference?

Both keep the transcript from overflowing, but compaction summarizes earlier turns into a condensed form (preserving gist, losing exact detail), while context editing removes stale content outright (old tool results, completed thinking). Use compaction when you need the substance of old turns; use editing when they're just dead weight. Many long-running agents use both.

How is memory different from the context window?

The context window is what the model sees in a single call — ephemeral, rebuilt each request. Memory persists across sessions: a file or store the agent reads and writes, surviving between conversations. You store durable facts in memory and load only the relevant ones into context when needed, rather than keeping all history in-window forever.

Why does my long agent get expensive so fast?

Because the transcript grows and every prior tool result is reprocessed on each call — cost scales roughly quadratically with length. The fixes are context engineering: cap and summarize tool output, enable compaction or context editing, cache the stable prefix, and push durable facts to memory. The framework you chose won't do this for you.