Shift Note: Moving the App to Its Next Model

Today’s change is to move a running app from its current AI model to another one. The app already serves real users, so the job includes protecting the behavior they depend on: useful answers, working integrations, acceptable response times, and predictable spending.

A model migration means changing the AI system behind a feature. That feature might summarize support tickets, extract fields from invoices, or help users write code. Your app sends instructions and data to a model, then uses its response. Replacing that model can change both the response and the assumptions your software makes about it.

For example, a team might evaluate moving a ticket summarizer from Gemini 2.5 Flash to DeepSeek V4 Flash. Those names identify the candidates; they do not establish which one fits the workload. The team needs evidence from its own tickets.

Start by writing down what the feature must keep doing. For a ticket summarizer, that could mean preserving the customer’s main problem, avoiding invented details, and returning fields your application can read. Add practical limits for response time and spending, based on the product’s needs.

Save the current setup before changing anything. Record the model identifier, instructions, request settings, and response handling. Keep credentials in your existing secret management system. This gives the team a known configuration to restore if the new model causes trouble.

Next, assemble a small but varied set of examples from real use, with sensitive information handled according to your organization’s rules. Include ordinary requests, long inputs, missing details, and cases that previously caused failures. Keep some examples separate from the ones used to adjust instructions, so your final check includes unfamiliar inputs.

• Check whether each answer is accurate and useful for the task your users are trying to complete.
• Check whether structured responses contain the required fields and can be parsed by your application.
• Check whether tool requests use valid arguments and respect the permissions enforced by your application.
• Measure response time and cost across representative requests, including retries and failed attempts.

An API connection that succeeds is only the first checkpoint. Two models can accept similar requests while following instructions differently. One may add explanatory text where your application expects a data object. Another may interpret an ambiguous instruction differently or choose to call a tool at a different moment.

Make instructions explicit where those differences matter. Describe the required output, define what to do when information is missing, and validate responses before using them. Keep authorization checks in application code, especially when a model can request actions such as changing records or sending messages.

Move traffic in stages once the candidate passes your checks. Begin with internal use or a limited share of suitable requests. Compare failures, answer quality, response times, and spending with the current setup. If you copy live requests to a second model for evaluation, account for the extra cost and confirm that sending the data there is permitted.

Prepare the return path before expanding the rollout. A configuration switch can make restoring the previous model straightforward, provided its integration still works. Define the conditions that should stop the rollout, and assign someone to make that decision. Avoid automatically repeating actions that may already have completed, such as creating an order.

Relay365 can fit when you want one pay-as-you-go relay desk for several model families. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Its API base is https://api.api365relay.com; OpenAI clients add /v1. A shared connection can simplify access, while each candidate still needs checks for the features and behavior your app uses.

If that access pattern suits your evaluation, create a Relay365 account and start with your saved test cases. After the rollout, leave a shift note recording the active model, the checks completed, any remaining limitations, and the route back. That record gives the next engineer enough context to operate the change.