Shift Note: Long System Prompts Keep Adding Input Cost

A system prompt is the standing set of instructions an application gives an AI model before it handles a user’s request. It might define the assistant’s job, its writing style, and the rules it must follow. Those instructions can be useful, but they also become part of the input the model processes. When the same long prompt accompanies thousands of requests, its cost deserves a place in the operating budget.

The standing instructions travel with each request

Models process text as tokens: small pieces of words, punctuation, and other content. A longer system prompt usually means more input tokens. For APIs that charge by token usage, those tokens contribute to the bill alongside the user’s message, conversation history, tool definitions, and other supplied material.

Here is a simple example. Suppose you remove 2,000 tokens from a system prompt used in 10,000 requests. That removes 20 million input tokens from the requests you submit. The money saved depends on the model’s input price and whether any of that text qualified for discounted caching. This is token arithmetic, not a promise of a particular saving.

Long prompts can also consume context capacity that could hold a document or useful conversation history. They may increase the time spent processing input, although the effect on response time depends on the model, caching, output length, and service conditions.

Trim repeated instructions before cutting useful rules

The easiest material to remove is often repetition. A prompt may ask the model to be concise in three different places, then add several paragraphs explaining what concise means. Another may contain rules copied from earlier experiments that no longer match the application.

• Combine repeated instructions into one clear sentence that preserves the intended behavior.
• Remove background information that does not affect the assistant’s decisions or answers.
• Delete obsolete rules and resolve contradictions instead of adding another exception.
• Keep examples that teach a difficult requirement, and remove examples that merely repeat an obvious instruction.
• Move task-specific reference material into the requests that need it, while retaining the rules required on every request.

For example, “Please always remember to make every response as brief as possible while still including the information needed to answer the user” can become “Answer concisely while preserving necessary detail.” The shorter version keeps the practical instruction. However, replacing a detailed refund policy with “Handle refunds appropriately” removes information the model needs.

Preserve operational boundaries. Rules about permissions, private data, required output structure, and when to seek human help often earn their space. Trim their wording carefully, then check that the assistant still follows them.

Caching changes the bill, but still needs checking

Some services support prompt caching, which can reduce the charge for eligible repeated input. A stable system prompt may benefit when it appears as an identical prefix across requests. Eligibility, minimum lengths, retention periods, and pricing vary by provider and model.

Keep reusable instructions stable where practical. Put changing details, such as the current question or a timestamp, after the reusable prefix when the API’s message structure allows it. Check the service’s documentation and reported usage to confirm that caching actually occurred. Sending identical text does not by itself establish that it received a discount.

Caching also means that removing a token from a frequently cached prompt may save less money than removing an uncached token. Compare the actual billed categories before estimating savings.

Check the shorter prompt against real work

Save the original prompt and run both versions against a small, representative set of requests. Include routine questions, ambiguous requests, required output formats, and situations where the assistant must respect a boundary. Compare answer quality, instruction following, input usage, and cost. Restore any detail whose removal causes a meaningful regression.

If you use several model families, Relay365 is a pay-as-you-go relay desk that can fit that workflow. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Create a Relay365 account if that setup suits your work, then evaluate your shorter prompt with the models you intend to use, checking current pricing and caching support before estimating the bill.