More context does not give an agent more access to information
A production engineering guide to full history, summarization, selective retrieval, and evaluating what an AI agent actually uses from context.

User preference: invoices need a PO number
...
87 later messages
...
Agent: I will send the invoice without a PO number.
The failure is easy to misdiagnose. The preference was in the conversation, so the agent had the context. It did not have usable access to it at the decision point.
Longer context windows make this distinction harder to ignore. A model can accept a record without reliably locating the relevant fact, deciding that it applies now, or grounding the next tool call in it. The input limit is capacity. Agent reliability depends on information access.
This changes how to think about memory. The engineering question is not "How much history should we retain?" It is "Which evidence must this model call see, in which representation, so it can make this decision correctly?"
A long window does not behave like a database query
A database query returns a result after a deterministic match procedure. A language model receives a sequence and generates the next token conditioned on it. The fact that an input fits in the context window means it can influence the output. It does not mean the model will recover that fact consistently under distractions, competing instructions, or a long trajectory.
Liu et al. tested this directly in "Lost in the Middle". Across multi-document question answering and key-value retrieval, moving the relevant information changed performance substantially. Models performed best when the relevant evidence was near the start or end of the input and worse when it sat in the middle, including models built for long contexts.
The result matters for agents because their context is rarely a curated document. It is an accumulation of user requests, tool observations, retries, intermediate plans, error messages, source passages, and policy text. Each new item can be useful. It can also change the retrieval burden inside the next model call.
Chroma's Context Rot technical report isolates this burden across 18 models. It finds performance degradation as irrelevant input grows, with factors such as semantic similarity, distractors, and haystack structure affecting the result. Its LongMemEval comparison is especially useful for agent engineers. Focused prompts averaged about 300 tokens. Full prompts averaged about 113,000 tokens and required the model to find relevant history before reasoning. Across models, the focused version performed better.
That is not evidence that every long-context application should retrieve a few hundred tokens. It is evidence that a model call which must both search a noisy history and solve a task has a harder job than one given the evidence already selected.
Preserve, make available, use correctly
Context systems are usually discussed as a choice between chat history, summaries, and vector search. That framing hides three independent failure points.
First, the system must preserve a fact. A summary can omit it. A store write can fail. A later update can overwrite it. Second, the fact must become available for the relevant call. Retrieval can miss it, rank it too low, or return an older version. Third, the model must use it correctly. It may misread the date, defer to a newer but less authoritative message, or ignore it while following an attractive tool result.
These properties imply different metrics.
| Property | Question | Useful measurement |
|---|---|---|
| Preservation | Did the source fact survive? | Exact fact recall after write, update, and compression. |
| Availability | Did the model receive the right evidence? | Retrieval recall at k, freshness, and source coverage. |
| Correct use | Did the decision depend on it correctly? | Output or trajectory check against a controlled counterfactual. |
This distinction is why token count is a poor top-line quality metric. A 100,000-token transcript can preserve the right fact and fail availability. A 300-token retrieved context can have high availability and still fail correct use because the model mistakes a historical preference for a current instruction.
Strategy 1 keeps the full conversation history
Appending every message has one major advantage. It preserves fidelity. The original wording, source order, tool output, and changes over time remain available. This is useful when the task needs an audit trail, exact quoted language, or dense local reasoning over recent work.
It also asks the main model to act as a retriever. On every call, the model has to distinguish the active plan from discarded attempts, old tool observations from current state, and durable facts from conversation noise. Latency and cost grow with the prompt. More subtly, the probability of a relevant but poorly positioned fact being overlooked rises as the history grows.
LongMemEval makes that failure concrete. The ICLR 2025 benchmark tests extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. It uses 500 curated questions embedded in scalable histories. The authors report a 30% accuracy drop for commercial assistants and long-context LLMs when memorizing information across sustained interactions.
Full history is still the right choice in some places. Keep the last several turns and the current tool interaction at high fidelity. A model needs the exact arguments and results of an in-flight action. Do not summarize an approval request into "approval pending" when the amount, recipient, and policy evidence determine whether it is safe to execute.
The mistake is treating full history as the default memory architecture. It makes one LLM call own persistence, relevance ranking, temporal interpretation, and reasoning at once.
Strategy 2 compresses history through summarization
Summarization trades fidelity for a bounded prompt. LangChain's SummarizationMiddleware monitors conversation size and can replace older messages with a summary while retaining a recent window.
from langchain.agents import create_agent
from langchain.agents.middleware import SummarizationMiddleware
agent = create_agent(
model="openai:gpt-5",
tools=tools,
middleware=[
SummarizationMiddleware(
model="openai:gpt-5-mini",
trigger=("tokens", 4_000),
keep=("messages", 12),
)
],
)
The current LangChain API also lets a team trigger on a fraction of the model window or a message count, combine conditions, customize the summary prompt, and choose how much recent history remains verbatim. These are policy choices, not tuning constants. A support assistant with short independent requests may compress aggressively. A procurement workflow may need to retain exact dates, amounts, approvers, and source IDs until the case closes.
Summaries fail in two ways.
They can remove an important fact. A short summary may retain "customer requested a refund" while dropping "only after delivery confirmation." They can also create a false durable fact. A model may summarize a tentative hypothesis as a decision, or retain a preference after the user has revoked it.
Avoid asking a prose summary to carry all durable state. Keep machine-critical facts in typed state or a store. A summary can explain the narrative around a workflow. State should hold the fields that must remain exact.
from typing import Literal, TypedDict
class CaseState(TypedDict):
case_id: str
identity_verified: bool
approval_status: Literal["not_requested", "pending", "approved", "rejected"]
approved_amount_cents: int | None
policy_version: str
running_summary: str
The summary is useful context. The typed fields are the control plane. Do not make a model recover approved_amount_cents from prose if an exact field exists.
LangChain's context-engineering model makes this separation explicit. Model context is transient. State is short-term, conversation-scoped memory. A store holds cross-conversation information. Lifecycle context covers middleware operations such as summarization and guardrails. LangGraph checkpointing persists state across a thread, which makes a workflow resumable without treating the chat transcript as its only database.
Strategy 3 retrieves selective evidence from external memory
Selective retrieval separates finding from reasoning. The system indexes history, documents, or structured records outside the current prompt, then retrieves a small set of evidence for the present task. This often lowers inference cost and lets the main model focus on reasoning.
It also creates a new failure mode. If the retrieval query is wrong, the relevant fact never reaches the main model. A model cannot correct a false negative that is absent from its context.
The LongMemEval authors frame long-term memory as three stages: indexing, retrieval, and reading. Their findings support details that many simple RAG implementations skip, including session decomposition, fact-aware index keys, and time-aware query expansion. The system needs temporal logic because "my current address" and "the address I gave last January" can both be correct records.
For operational agents, retrieval should return evidence with source metadata, timestamps, and authority rather than raw text alone.
{
"fact": "Invoices require a PO number",
"source_type": "customer_policy",
"source_id": "policy-94",
"valid_from": "2026-08-01",
"supersedes": "policy-71",
"authority": "account_owner",
"retrieval_score": 0.92
}
retrieval_score is not a permission check. It is an input to ranking. The model or application still has to resolve whether the fact applies to this customer, time, and operation. Retrieval quality needs relevance metrics, but production reliability also needs freshness, authority, and contradiction checks.
The engineering advantage of retrieval is not simply fewer tokens. It gives each subsystem a narrower responsibility. The retriever finds candidates. State supplies current workflow facts. The model compares the resulting evidence and plans the next action. An evaluator can then locate whether a failure started in indexing, retrieval, or reasoning.
Context assembly belongs in middleware
The best context often depends on runtime facts that the model should not be asked to infer. Tenant, role, current workflow state, attachment IDs, and permission scope are known by the application. Use middleware to read these values and assemble the model request deterministically.
from collections.abc import Callable
from langchain.agents.middleware import ModelRequest, ModelResponse, wrap_model_call
@wrap_model_call
def add_case_context(
request: ModelRequest,
handler: Callable[[ModelRequest], ModelResponse],
) -> ModelResponse:
case = request.state.get("case")
evidence = retrieve_case_evidence(
case_id=case["case_id"],
query=request.messages[-1].content,
limit=5,
)
context_message = {
"role": "user",
"content": render_evidence(evidence, case),
}
return handler(request.override(messages=[*request.messages, context_message]))
The example is deliberately incomplete. retrieve_case_evidence needs its own timeout, authorization filter, and evaluation suite. render_evidence needs a stable format that differentiates source text, current state, and untrusted user content. Middleware is the point where these contracts can be applied before the LLM sees the prompt.
Avoid injecting every retrieval result because a retriever returned it. Set a token budget by evidence class. Prefer current policy records over semantically similar old ones. Preserve source identifiers so a later tool call can read the authoritative record rather than acting on a compressed paraphrase.
Staleness is a temporal access problem
Teams often call stale context a retrieval problem. It is usually a data-model problem first.
An embedding index can return a highly similar record that is no longer true. A conversation summary can retain a decision that was later reversed. A full transcript can contain both versions without stating which one controls. The agent needs validity intervals, versioning, provenance, and a conflict policy.
For mutable facts, store the current value separately from historical events. Retrieve history only when the task needs a comparison or explanation. For a user preference, record who set it, when it became effective, and whether a later event supersedes it. For a policy, retrieve the active version by date and scope before giving semantic similarity any authority.
This is also where tool design matters. A search_policies tool that returns all near matches is useful for research. A get_active_policy tool with a date and customer scope is safer for execution. The latter moves temporal selection into code.
Measure the whole information path with LangSmith
Context changes are behavior changes. A shorter prompt can make latency better while silently removing the fact that prevents an invalid action. A more aggressive retriever can improve recall while adding distractors that reduce reasoning accuracy. Token counts alone cannot tell you which trade-off won.
LangSmith supports the loop needed to test these strategies. Its evaluation workflow supports datasets, code evaluators, human review, LLM judges, offline experiments, and online monitoring. Use one dataset across full-history, summarization, and retrieval variants. Compare the same cases rather than comparing anecdotal traces.
Build cases around context failures.
{
"case_id": "policy-reversal-027",
"history": "... customer waives PO requirement ... later revokes waiver ...",
"request": "Draft the invoice for the current order.",
"expected_context_facts": ["PO required", "waiver revoked"],
"expected_action": "request_purchase_order",
"forbidden_action": "send_invoice"
}
Then score each stage.
| Metric | Full history | Summary | Retrieval |
|---|---|---|---|
| Exact fact survival | Does it remain in the transcript? | Did compression retain it accurately? | Did indexing write the fact? |
| Evidence availability | Did the model locate it? | Was it kept or reconstructed? | Was it retrieved within the budget? |
| Freshness | Did later turns override it? | Did the summary reflect the update? | Did the current record outrank the old one? |
| Decision grounding | Did the action follow the fact? | Did the action follow the fact? | Did the action follow the fact? |
| Cost and latency | Input growth per turn | Summary calls plus input | Retrieval plus smaller input |
Run counterfactual pairs. Keep the task fixed, change only the position of a fact, its timestamp, or whether a distractor appears. This reveals whether an agent used the evidence or merely produced a plausible answer. Add failed production traces to the dataset. A memory defect should become a permanent regression test.
Choose the smallest context that still contains the evidence
Full history is useful for short sessions and exact local reasoning. Summaries work when older narrative context matters but every detail does not. Retrieval is usually necessary when history spans many sessions or an agent needs durable, searchable knowledge. Production agents often combine all three.
Keep a recent high-fidelity window. Store exact workflow state separately. Retrieve a compact set of authorized, current evidence. Summarize the older narrative only after extracting facts that need exact storage. Then test whether the agent can still act correctly when information position, noise, and history length change.
The important architectural shift is to stop measuring memory by what the system saved. Measure it by what the model had available and used at the point of action. Context engineering is an information-access discipline. Token management is one constraint inside it.
Sources
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2024.
- Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025.
- Hong, Troynikov, and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma technical report, 2025.
- Liu et al., Context as a Tool: Context Management for Long-Horizon SWE-Agents, Findings of ACL 2026.
- LangChain, context engineering, prebuilt middleware, and LangSmith evaluation.

