Shift note: stop paying full price for repeated prompts

Repeated input is the first place to inspect

A support assistant may send the same policy manual with every question. A coding assistant may repeatedly include project instructions and tool definitions. Those repeated input tokens are where prompt caching can reduce API costs.

Prompt caching lets a provider reuse processing associated with eligible input it has already seen. Depending on the service, you may receive a lower price for cached input, faster processing, or both. It does not mean the model reuses an old answer: the new request still generates a response.

The practical goal is to make useful context reusable, then verify that the billing reflects that reuse.

Put the stable material before the changing question

Many caching systems recognize a matching prefix: the beginning of a request that stays the same across calls. Put durable instructions, reference documents, examples, and stable tool definitions first. Put the user’s changing question and request-specific details afterward.

Imagine an assistant using Gemini 2.5 Flash to answer questions about a product manual. Keeping the same manual before each new question creates a potential caching opportunity, subject to the provider’s caching requirements. Placing a changing timestamp before the manual can prevent an otherwise reusable prefix from matching.

• Keep shared instructions and documents in a consistent order.
• Move timestamps, request identifiers, and other changing fields after reusable context whenever the application allows it.
• Keep tool definitions stable when the available tools have not changed.
• Version shared instructions deliberately so you can explain changes in cache usage.

Do not preserve outdated instructions merely to protect a cache hit. Correct answers remain the priority.

Check the cache rules on the route you use

Caching behavior varies by model, provider, and API route. Some services cache eligible input automatically. Others require explicit cache creation, cache references, or markers. Minimum input lengths, retention periods, and supported content can also differ.

Before changing your application, check whether your chosen route supports caching, how you enable it, and which usage fields identify cached tokens. An OpenAI-compatible request format alone does not establish that every provider’s caching controls are supported.

Relay365 is a pay-as-you-go relay desk for many model families and can fit a workflow that accesses several providers through one service. For caching decisions, confirm the behavior and pricing of the specific model route you intend to use.

Count writes and storage before calling it a saving

A cache hit is useful only when the total cost improves. Depending on the service, creating a cache can carry a different input rate, and retaining it can incur storage charges. A large document used once may cost less without explicit caching.

Compare two totals for the same workload. First, estimate ordinary input and output charges without caching. Then add cache creation or write charges, cached reads, uncached input, storage where applicable, and output charges. Prompt caching generally does not discount newly generated output.

For illustration, suppose normal input for a repeated document costs one billing unit per request, a cache write costs two units, and each later cached read costs 0.2 units. Ten requests would cost 3.8 input units with caching instead of ten, before any storage charges. These are invented arithmetic values, not provider prices.

Measure a real repeat workload before expanding

Start with one task that repeatedly uses substantial shared context. Run a representative sequence of requests and record reported cached input, uncached input, output, and total charges. Include requests after idle periods to see whether retention fits your traffic.

A high cache hit rate can still produce a disappointing bill if output dominates spending or you add unnecessary text to qualify for caching. Keep prompts useful and concise, then optimize the repeated material that remains.

If Relay365 fits your workflow, create an account and test a small repeat workload on your chosen route. Expand only after usage records and charges show that the caching arrangement saves money for your actual traffic.