Shift Log: Prompt Cache Winners and Cost Traps

The cost-saving question on this shift is not whether a cache reports hits. It is whether those hits reduce the bill for useful, correct responses. Automatic prefix caching, explicit cache management and application-level response caching target different costs. Treating them as interchangeable is how a promising optimization becomes another line item.

Automatic prefixes beat manual controls for steady traffic

Automatic prefix caching is the lowest-effort starting point when the provider supports it. Put stable instructions, tool definitions and reference material first; put user-specific content last. Repeated leading tokens may then qualify for discounted processing under the provider’s rules.

Its opponent is explicit caching: supported APIs let you create or designate reusable cached content, sometimes with retention controls. That gives you more lifecycle control, but can introduce cache-write charges, storage charges and cleanup work. Availability and billing vary by endpoint and model.

For a busy support assistant with a stable policy document, automatic caching usually deserves the first trial. For a scheduled document-analysis batch with predictable reuse, explicit caching may offer better control. Neither wins merely because its dashboard shows more hits.

Verdict: Start with automatic caching for steady, repetitive traffic. Choose explicit management when predictable reuse justifies its setup and retention costs.

Stable serialization beats aggressive prompt trimming

Removing unnecessary tokens saves input cost whether caching works or not. But constantly rewriting a shared prefix can sacrifice reuse. A smaller prompt that changes every request can cost more than a slightly larger, stable one.

Keep timestamps, request IDs and personalization outside the shared prefix. Serialize tool definitions deterministically, preserve document ordering and version shared instructions deliberately. Match the provider’s prefix rules rather than assuming semantically equivalent text will count as identical.

Do not preserve obsolete instructions just to retain cache hits. First remove genuinely unnecessary content; then stabilize what remains. If an agent appends conversation turns, keep earlier turns unchanged where practical, while applying your normal context and safety limits.

Verdict: Trim waste once, then preserve the useful prefix. Constant cosmetic editing is a poor caching strategy.

Response caching wins only when answers are reusable

Provider-side prompt caching reuses input processing; the model still generates a response. Application-level response caching can avoid the model request entirely. For identical, nonpersonalized questions against unchanged data, that is often the stronger cost lever.

The trade-off is correctness. A cached answer can become stale, cross permission boundaries or miss a changed instruction. Its key should capture relevant document versions, tenant scope, permissions, model configuration and other answer-changing inputs. Similarity-based reuse requires additional evaluation because similar questions need not have interchangeable answers.

For a shared reference lookup, response caching may win. For personalized advice or live account status, prompt caching is usually the safer optimization because it does not reuse the final answer.

Verdict: Cache responses only when equivalence and freshness are enforceable. Otherwise, cache the shared input.

Measured reuse beats longer retention and wider routing

Longer retention helps only when requests return before expiry and avoided processing costs exceed any additional charges. Bursty workloads may benefit from warming a cache before a batch; sporadic workloads may never recover the write or storage cost.

Routing complicates that calculation. Do not assume caches follow requests between providers, accounts, regions or model deployments. A cheaper route with no reusable prefix may still win, while frequent switching can undermine an otherwise economical cache.

• Compare total cost per successful task, including input, output, cache writes, reads, retention and retries.
• Measure cache-eligible tokens separately from actual cached tokens, using billing records where available.
• Test with representative traffic gaps and invalidate caches when source data or permissions change.

The desk recommendation is to verify the billing path

Relay365 fits teams that want pay-as-you-go access across model families through a relay desk. It works with Claude Code, Codex, Gemini CLI and OpenAI SDK clients. Before relying on caching, verify that your selected route exposes the required cache controls, usage fields and billing treatment; client compatibility alone does not establish cache support.

Create a Relay365 account if that access model fits your workflow, then run a small billed comparison. The final winner is the strategy that lowers cost per correct task—not the one with the longest retention or prettiest hit-rate chart.