Shift Note: Choosing a Model for the Support Queue

A customer support bot needs to answer ordinary questions reliably, recognize when it lacks information, and hand difficult cases to a person. Picking its model starts with the work arriving in your queue. A polished demo offers limited evidence about whether the bot can handle a confusing refund request during a busy shift.

The model is the part that interprets a message and generates a response. Its results also depend on the instructions, information, and tools you give it. Treat your selection as a small operational trial with clear acceptance checks.

Start with the tickets the bot will actually receive

Choose a narrow first assignment, such as explaining delivery policies or helping customers find invoices. Answering from a help article requires different capabilities from investigating an account or deciding whether an exception applies.

Collect a representative sample of past tickets, removing personal information before using them for evaluation. Include clear questions, vague requests, follow-up messages, and cases that needed human judgment.

• Identify which questions the bot should answer using approved documentation.
• Identify which requests require current account information or an authorized action.
• Identify which situations require a person, including policy exceptions and unresolved complaints.

This gives you a practical definition of success before model names and pricing influence the decision.

Give the model access to the right support information

A model should not guess your return window or invent an order status. Supply current policies and retrieve the relevant help articles for each question. Retrieval simply means finding useful information and giving it to the model before it answers.

Account questions need a connection to your business systems. Your application should verify the customer's identity and control which records and actions are available. The model can request an order lookup, but your software must decide whether that lookup is allowed.

Keep these connections consistent when comparing models. Otherwise, a better answer might come from better source material rather than a better model.

Compare two candidates on the same support cases

Start with a small shortlist. For example, you could evaluate Claude Haiku 4.5 and GPT-6 Astra using the same instructions, documents, and tickets. Treat both as candidates whose suitability needs testing in your environment.

Have a support agent review the answers against a written checklist. A friendly tone matters, but it cannot compensate for an incorrect refund promise.

• Check whether each answer agrees with the approved policy and available account facts.
• Check whether the bot asks a useful clarifying question when information is missing.
• Check whether it follows the conversation across several messages.
• Check whether it escalates appropriately and gives the agent a useful summary.
• Measure how long customers wait for a usable answer.

Include messages that ask the bot to ignore its instructions or disclose another customer's information. Use the results to improve application controls as well as model selection.

Measure the cost of reaching a useful outcome

API pricing often uses tokens, which are pieces of text processed or generated by the model. Longer conversations, repeated instructions, and large document extracts can increase usage.

Compare the total model cost across your test cases alongside the quality of the results. A cheap reply can become expensive if it causes repeated questions or avoidable agent work. A more expensive model also needs to earn its place through better outcomes on your actual tickets.

You may eventually route straightforward questions to one model and difficult cases to another. Start with a single model if that makes the system easier to understand and maintain.

Keep the first rollout easy to inspect

Relay365 fits teams that want pay-as-you-go access to several model families through one relay service. It works with OpenAI SDK clients, which can make testing different candidates more convenient. Check each candidate's supported features and behavior through your intended integration.

Once your evaluation cases are ready, create a Relay365 account to run your shortlist through the same support workflow.

Begin with a limited rollout and review conversations regularly. Track wrong answers, unnecessary escalations, missed escalations, and customer follow-up. Choose the model that meets your written checks, then repeat the evaluation when policies, tools, or model versions change.