Caching Strategies for LLM Agents: A Production Guide

LLM agents make caching unusually valuable and unusually dangerous. A single request may send thousands of prompt tokens to a model, retrieve documents, call paid APIs, and perform several seconds of orchestration. Reusing safe work can remove most of that latency and cost. Reusing the wrong work can leak one tenant's data, serve an answer based on obsolete documents, or repeat a side effect such as sending an email.

The practical solution is not one magical cache. Production agents usually use several narrow caches, each placed around a specific kind of work and governed by an explicit freshness contract. Stable prompt prefixes are reused by the model provider. Deterministic reads use exact keys. Retrieval and tool calls have source-aware entries. Similarity-based reuse is limited to low-risk domains. Actions use idempotency records rather than ordinary response replay.

This guide explains the ten strategies developers encounter most often, how they fit together, and where each can fail.

Layered caching avoids agent and tool costs when an earlier safe layer can answer.

Why Agent Caching Is Different

A conventional HTTP response often depends on a route, query parameters, identity, and a database version. An agent response may additionally depend on the entire conversation, system instructions, model revision, temperature, tool definitions, retrieved chunks, current time, and intermediate tool results. Two requests with the same latest user message are therefore rarely equivalent.

Start by classifying operations as reads or actions. A read observes state: searching documents, fetching weather, parsing a file, or asking a deterministic model to format known data. It may be cached when the key captures every behavior-changing input and the freshness policy matches the source. An action changes state: sending a message, creating an issue, charging a card, or writing a record. Its result can be recorded, but a retry must be deduplicated with an idempotency key. Returning a cached sentence that says “done” is not proof that the action happened, and replaying the tool call may happen twice.

A second distinction is reuse scope. Some entries are global because their input is public and immutable. Others must be isolated by tenant, user, authorization scope, region, or data residency boundary. If two callers could legitimately receive different answers, they cannot share an entry unless the distinguishing attributes are part of the key and storage policy.

Finally, define freshness before implementation. “Cached for five minutes” is a mechanism, not a contract. Ask how stale the result may be, which event invalidates it, whether stale data can be served during an outage, and what the system does when validation is uncertain. Correctness-sensitive paths should bypass an unavailable cache rather than trust an ambiguous entry.

The Common Strategy Catalog

1. Prompt prefix caching

Prompt prefix caching reuses computation for the stable beginning of a model request. The reusable prefix commonly contains the system prompt, tool schemas, policy text, and long reference material. Dynamic conversation turns follow that prefix. Many model providers implement this internally and expose cached-input usage or discounted token accounting.

This is the first optimization to consider when every call carries a large, repeated instruction block. It changes neither the model answer nor application semantics; it avoids recomputing attention state for identical leading tokens. It also has fewer application-level invalidation problems because a changed byte naturally creates a different prefix.

Structure messages deliberately: stable content first, volatile content last. Avoid timestamps, request IDs, randomized tool ordering, or per-user text inside an otherwise shared prefix. Track the exact serialized bytes, model revision, and provider cache policy. A visually identical prompt can miss if whitespace, ordering, or tokenization differs.

The primary failure mode is disappointing hit rate, not incorrect reuse. Prefix caching is usually provider-scoped, short-lived, and unavailable across models or regions. Measure cached input tokens rather than assuming that a repeated logical prompt is a byte-identical prefix.

2. Cache-aside

Cache-aside is the most common application pattern. The caller checks the cache, computes on a miss, stores the result, and returns it. The cache does not own the source of truth; the agent service owns lookup and population.

value = cache.get(key)
if value is missing:
    value = compute()
    cache.set(key, value, ttl)
return value

Cache-aside keeps policy in the application and works around models, tools, retrieval, and parsing.

Its strength is control. Each call site can define key fields, TTL, negative caching, serialization, and whether stale data is acceptable. It also degrades naturally: when the cache is unavailable, a read can compute from the source.

Its weakness is duplicated policy and cache stampede risk. Many workers may observe the same miss and launch the same expensive model or tool call. Use single-flight locking, request coalescing, or probabilistic early refresh for hot keys. Locks need bounded leases and fencing; an abandoned lock must not block the key forever. Do not let cache failure become application failure unless the cache itself is the authoritative idempotency store.

3. Exact-match response caching

Exact-match response caching hashes the complete normalized model request and stores the response. A safe key includes the model and revision, system and conversation messages, tool schemas, generation parameters, prompt version, tenant boundary, relevant source versions, and response format. This is effective for deterministic, read-only calls such as classification, extraction, normalization, or templated explanation.

Use conservative normalization. Removing irrelevant transport metadata is useful; changing message order, collapsing meaningful whitespace, or ignoring a tool version is unsafe. Temperature zero improves repeatability but does not make two different contexts equivalent. Model providers may update serving behavior behind a name, so pin or include the resolved revision when reproducibility matters.

Exact caching has a lower hit rate than semantic caching, but its correctness argument is much stronger: equal keys represent equal declared inputs. Store provenance beside the value—creation time, model, prompt version, source hashes, and policy version—so operators can explain a hit and invalidate a namespace.

Do not cache responses that depend on current time unless the time bucket is explicit. Do not cache hidden authorization decisions into a globally shared key. Never use the latest user message alone as the key for a conversational agent.

4. Tool-result caching

Tool-result caching wraps expensive read-only tools: search APIs, database reports, package metadata, geocoding, document parsing, or repository analysis. This layer often saves more predictably than final-response caching because tool inputs are structured and source freshness is easier to define.

Build the key from the tool name and version, canonical arguments, caller scope, source version, and environment such as region or locale. Validate tool arguments before lookup so two semantically different inputs are not normalized together. Cache only successful results by default. Negative caching can protect an unavailable dependency, but use a short TTL and distinguish “not found” from timeout, permission denial, and server failure.

A tool that mixes reads and writes must expose separate operations. Never place a generic cache decorator around a function that might send, create, delete, or mutate based on an argument. If a read result embeds short-lived credentials or signed URLs, either exclude it or key and expire it according to those credentials.

Useful metrics include hit rate per tool, avoided API calls, source age at hit time, error-cache rate, and refresh latency. Event-driven invalidation is ideal when the underlying system emits reliable change notifications; otherwise combine a bounded TTL with source version checks.

5. Retrieval caching

Retrieval-augmented generation has several independently cacheable stages: query embedding, candidate vector search, document loading, reranking, and final context assembly. Caching the final answer alone hides these opportunities and couples reuse to model wording.

Embedding cache keys include the normalized query, embedding model revision, dimensions, and preprocessing version. Vector-search keys additionally include index version, filters, top-k, distance metric, and tenant scope. Reranker keys include the ordered candidate IDs, candidate content hashes, reranker revision, and scoring parameters. Loaded immutable chunks can be content-addressed by hash and retained for a long time.

Index freshness is the central risk. A query result from yesterday may omit a document added today even though every returned chunk still exists. Include an index generation or corpus snapshot in the key. For rapidly changing corpora, use event invalidation or a short TTL. Security filters belong in the retrieval operation and the key; filtering cached global candidates afterward can still leak metadata or produce different ranking behavior.

Cache embeddings broadly when inputs are public and the model is fixed. Cache retrieval results more narrowly. Preserve document IDs, versions, scores, and retrieval time as provenance so an answer can cite the exact evidence used.

6. Semantic caching

Semantic caching embeds a new request, searches prior requests, and reuses a response when similarity exceeds a threshold. It raises hit rate because “How do I reset my password?” and “I forgot my password” can share an answer. It also creates the hardest correctness problem: similarity is not equivalence.

A high cosine score can hide different dates, quantities, jurisdictions, accounts, permissions, or negation. “Cancel my transfer” is close to “confirm my transfer” while requiring opposite actions. Treat every semantic false positive as a product defect, not a harmless approximation.

Use semantic caching only for low-risk, read-only, relatively timeless content with a validated intent boundary. Partition by tenant, locale, policy version, and authorization scope before vector search. Apply exact filters for entities and constraints, then a conservative threshold. Store the matched query and similarity score as provenance. Sample hits for review and maintain an immediate kill switch.

Do not use it for actions, personalized financial or medical guidance, precise calculations, rapidly changing operational data, or answers whose evidence set must be current. Shadow mode is the safest rollout: compute the normal answer, record what the cache would have returned, and compare before enabling reuse. A semantic false positive rate averaged across all traffic can conceal severe errors in a small critical intent, so report by domain and intent.

7. TTL caching

TTL caching expires entries after a fixed duration. It is simple, bounded, and appropriate when acceptable staleness is naturally expressed in time: seconds for inventory, minutes for search results, hours for public metadata, or effectively forever for content-addressed immutable documents.

Choose TTL from the business freshness requirement, not from infrastructure convenience. Add jitter so popular keys do not expire simultaneously. A soft TTL may mark an entry stale but still servable; a hard TTL forbids serving it. Record creation and source timestamps because remaining TTL alone does not tell an operator how old the underlying data was when cached.

TTL is not enough for urgent revocation. Permission changes, deleted content, safety-policy updates, and corrected facts may require event invalidation or versioned keys. Conversely, an extremely short TTL can create a permanent miss storm without materially improving correctness. Measure age-at-hit distributions and refresh pressure to tune it.

8. Versioned caching

Versioned caching places behavior versions directly in the key or namespace. Deploying prompt:v8, tools:v3, or corpus:2026-08-16 makes old entries unreachable without scanning and deleting them. This is the safest default invalidation technique for immutable releases.

Version every input whose change can alter the answer, then hash the canonical representation.

Versions should be derived from deploy artifacts or content hashes when possible, not manually remembered integers. A canonical key might be:

agent:v4:{tenant}:{authz_hash}:{model_revision}:{params_hash}:
{prompt_hash}:{tool_schema_hash}:{corpus_generation}:{request_hash}

Keep human-readable prefixes for operations and hash high-cardinality or sensitive material. Do not place raw secrets, prompts, or personal data in observable key names. Namespace retirement can remove old entries later, but correctness does not wait for deletion.

The failure mode is missing a version dimension. A tool can keep the same name while changing semantics; a corpus can update without its generation advancing. Define ownership for every version and test that deployments actually change the expected namespace.

9. Stale-while-revalidate

Stale-while-revalidate serves a recently expired value immediately and refreshes it asynchronously. It reduces tail latency and prevents many callers from waiting behind the same refresh. Entries usually have a fresh window, a stale-but-servable window, and a hard expiry.

Serve stale data only inside an explicit safety window; refresh once in the background.

This strategy works for public documentation, recommendations, search results, and dashboards where brief staleness is acceptable. It is inappropriate for authorization, revocation, balances, one-time credentials, or safety rules. The response should carry age or provenance when users or downstream systems need to reason about freshness.

Use single-flight refresh so only one worker updates a hot key. If refresh fails, retain the stale value only until the hard deadline and expose the failure in metrics. Never silently extend stale life forever. Consider backoff to avoid hammering a failed source, but keep the correctness deadline independent from retry policy.

10. Idempotency records

Idempotency records protect actions from duplicate execution. The client or orchestrator supplies a unique operation key bound to the authenticated actor and canonical action parameters. The service atomically creates a record, executes once, and stores the durable outcome. A retry with the same key and parameters returns that outcome; the same key with different parameters is rejected.

Action safety requires an authoritative operation record, not a cached claim that an action succeeded.

Model the record explicitly: pending, succeeded, or failed, plus actor, parameter hash, timestamps, and external operation ID. Atomic creation is essential. Two workers must not both observe absence and proceed. Recovery for a stuck pending operation depends on the external system: query by external idempotency key, resume safely, or escalate for reconciliation.

Retention should cover the maximum retry and reconciliation window. Do not evict records merely because a general cache is under memory pressure. For high-value actions, store them in a durable transactional database rather than an ephemeral cache. This pattern also gives the agent truthful status: it can distinguish accepted, completed, failed, and still in progress.

Safe Cache Keys

A cache key is a compact statement of equivalence. If two operations share a key, the system asserts that reusing one result for the other is safe. Construct keys from canonical, validated data and include every dimension that may change the result:

• tenant, user, role, and authorization policy hash;
• model provider, model revision, and generation parameters;
• system prompt, conversation or task state, and response schema versions;
• tool names, schemas, implementations, and canonical arguments;
• retrieval index generation, filters, chunk versions, and reranker revision;
• locale, region, feature flags, safety policy, and relevant time bucket;
• normalized request content or its cryptographic hash.

Use SHA-256 or another collision-resistant digest for composite content. Keep a short namespace prefix for observability. Store provenance as value metadata rather than trying to decode every decision from the key. Sensitive inputs should be hashed with appropriate threat modeling; an unsalted hash of low-entropy personal data can still be guessed.

Authorization deserves special treatment. A role name may not fully describe row-level grants, document ACLs, or delegated scopes. Prefer a stable hash of the effective permission set or key by identity when computing that hash is unreliable. Invalidation must follow revocation latency requirements. A five-minute response TTL is unacceptable if access must disappear immediately.

Invalidation and Failure Modes

Use multiple invalidation mechanisms because they solve different problems. TTL bounds ordinary staleness. Versioned namespaces handle deployments and corpus generations. Event invalidation handles urgent source changes. Content addressing makes immutable transformations permanent. Stale-while-revalidate improves latency only where stale data remains safe.

Protect hot misses with request coalescing. Without it, an expired popular key can trigger a cache stampede: hundreds of identical model calls arrive simultaneously, increasing cost and delaying the refresh they all need. One worker should compute while others wait briefly, receive stale data when allowed, or fail according to the endpoint contract. Add TTL jitter and early refresh to avoid synchronized expiration.

Treat cache contents as untrusted input. Validate schemas during deserialization, authenticate shared cache connections, encrypt sensitive values where required, and restrict administrative access. Cache poisoning can originate from compromised tools, overly broad write permissions, or keys that omit attacker-controlled distinctions. Never deserialize executable objects from a shared cache.

Plan cache outages explicitly. For ordinary read optimization, fail open by bypassing the cache and using the source, with rate limits to protect dependencies. For idempotency records, failing open could duplicate an action; fail closed or use the durable source of truth. For uncertain authorization data, bypass to the authority rather than serving stale grants.

Observability and Rollout

Measure each layer independently. Report lookup volume, hit rate, miss reason, latency saved, model tokens avoided, external calls avoided, age at hit, eviction rate, refresh failures, lock contention, and cache stampede frequency. A high aggregate hit rate can hide a useless expensive layer, while a modest tool-cache hit rate may save most of the cost.

Attach provenance to traces: cache namespace, key fingerprint, entry age, source generation, hit or miss, and bypass reason. Do not log raw sensitive keys or values. For semantic caching, record similarity, matched intent, and sampled correctness labels. Track semantic false positive incidents separately from ordinary stale results.

Roll out from safest to riskiest. Enable provider prefix caching first, then deterministic tool and retrieval transformations, then exact-match model calls. Add stale serving only for endpoints with written freshness budgets. Evaluate semantic caching in shadow mode and enable it for narrow intents after review. Introduce idempotency before allowing agents to retry actions automatically.

Define kill switches by layer and namespace. When a prompt, tool, or corpus defect appears, operators should disable reuse or advance a version without deploying new application code. Capacity planning must include miss storms: the source system should survive a cold cache, or traffic must be shed deliberately. A cache is an optimization until the architecture quietly depends on it; document whenever it becomes authoritative.

The boring baseline is strong: provider prefix reuse, cache-aside around deterministic reads, exact versioned keys, source-aware TTLs, single-flight refresh, and durable idempotency for actions. Add semantic reuse only when measured repetition justifies its larger correctness surface.

EOF · cd ~