Shift Note: Test Two Models on Last Week’s Tickets

A new model is available, and the team wants to know whether to switch. The useful question is not which model has the most impressive demo. It is which one handles your actual work well enough, at an acceptable cost and response time. A small evaluation set gives you a repeatable way to answer that question.

Start with work that reached your desk

An evaluation set is a collection of inputs with a clear description of what a good answer should do. For a support assistant, that might mean customer questions, relevant account details, and the policy passages needed to respond. For a coding assistant, it could mean bug reports paired with tests that verify the fix.

Begin with a few dozen examples you can inspect carefully. That is a starting point, not proof that the results represent every future request. Remove secrets and personal information, and make sure you have permission to use the material.

• Include common requests so the test reflects everyday workload.
• Include difficult cases, such as missing context, conflicting instructions, and requests the assistant should decline.
• Include expensive failures, such as inventing a refund policy or changing unrelated code.
• Keep a separate, untouched group of examples for checking changes after you tune prompts.

Write the pass conditions before running models

Do not define success after seeing which answer you prefer. Write a simple scoring rubric first. For a support reply, you might require the correct policy, a useful next step, no invented account facts, and an appropriate tone.

Some checks can be automatic. A structured response can be tested for valid fields, and generated code can run against tests. Other checks need human review. A fluent answer can still be wrong, so do not let polished wording stand in for correctness.

Use a small scale, such as pass, partial, and fail, with a short explanation for each grade. Mark critical failures separately. An average score should not hide a model that occasionally exposes sensitive information or recommends an unauthorized action.

Give both candidates the same working conditions

You could compare Claude Haiku 4.5 and Gemini 2.5 Flash on the same ticket set. Treat those names as candidates, not predictions about which will win. Send each model the same task instructions, source material, and tool access.

Record the exact model identifier, prompt version, generation settings, output limit, and test date. Where settings differ across providers, document the difference rather than assuming identical parameter values create identical behavior. If you tune prompts for each candidate, use a comparable tuning budget and check the results on your untouched examples.

Repeat at least the important cases because outputs can vary. Hide model names during human scoring where practical, and shuffle the answer order to reduce preference bias.

Log quality beside cost and waiting time

Keep one result row per example and run. Record the grade, failure reason, input and output usage, reported charge, and elapsed time. If users see streamed responses, distinguish time to first visible output from time to completion.

Compare response times across the batch, not just the fastest request. Keep errors and retries in the log; excluding them makes the workflow look better than it felt. Estimate cost per successfully completed task as well as cost per request. A cheaper call may not be cheaper overall if it needs repeated attempts or substantial editing.

Use one access point without assuming identical models

Relay365 is a pay-as-you-go relay desk for many model families, making it a practical option for running candidates through one access point. OpenAI SDK clients use https://api.api365relay.com/v1 as the API base. Check supported model identifiers and features before testing; a shared interface does not make every model’s capabilities identical.

If that setup fits your workflow, create a Relay365 account and start with a bounded evaluation budget.

Leave a decision the next shift can repeat

Finish with a short decision record: which candidate met your requirements, which failures remain, and what would trigger another test. You may choose one model for routine tickets and another for harder cases. Keep the examples, rubric, prompts, and logs together so the next comparison starts from evidence rather than memory.