Picking a Model for Your Customer Support Bot

Choosing a model for a customer support bot is less about finding the most impressive demo and more about finding a reliable fit for your workload. Your bot needs to answer from approved information, recognize its limits, and hand customers to a person when necessary. Follow these steps to turn a broad model search into a manageable decision.

#1 · Define the bot’s first job

Start with a narrow scope, such as explaining shipping policies, troubleshooting account access, or answering product questions. Write down what the bot may answer, which systems it may access, and which requests require human help. A bot that explains a refund policy has different requirements from one that can issue a refund.

Set boundaries before comparing models. Otherwise, a fluent answer can look successful even when it promises something your business cannot deliver.

#2 · Build a small, realistic test set

Collect representative support questions, removing personal information before using them for testing. Include ordinary requests and awkward cases: vague descriptions, angry customers, missing order details, conflicting policy documents, and requests outside your scope.

For each case, record the expected behavior rather than one exact sentence. A successful response might explain the policy, ask a clarifying question, or escalate without guessing. Keep some cases separate from prompt development so your final comparison tests more than memorized examples.

#3 · Make the information setup consistent

Give each candidate the same instructions, relevant help-center excerpts, conversation history, and tool definitions. For changing facts such as order status or inventory, use authorized systems rather than expecting the model to know them.

If you use retrieval, check whether it supplies the correct documents. A poor answer may come from missing evidence rather than a weak model. Keep the setup stable so you can distinguish model quality from retrieval or integration problems.

#4 · Compare two candidates before expanding

Begin with two models, not a dozen. For example, you could evaluate Claude Haiku 4.5 and Gemini 2.5 Flash on the same support cases. Treat these as candidates to measure, not as promises of particular speed, price, or quality. Check current availability, pricing, and supported features before testing.

Relay365 fits naturally here if you want to compare model families through a pay-as-you-go relay desk. A shared access layer can simplify experimentation, but still verify each candidate’s tool calling, structured output, and other required behaviors.

#5 · Score outcomes, not writing style

Use a simple evaluation sheet that separates useful behavior from polished phrasing:

• Correctness: Does the response match the supplied policy and customer context?
• Grounding: Does it avoid inventing account details, exceptions, or commitments?
• Resolution: Does it answer the question or identify a useful next step?
• Escalation: Does it recognize when human judgment is required?
• Safety: Does it respect privacy and resist instructions hidden in customer messages or retrieved content?

Review consequential failures separately. An unauthorized refund promise should not disappear inside a favorable average score.

#6 · Measure the whole request path

Measure response time and cost using realistic conversations, including long histories, retrieved documents, tool calls, and retries. Token prices alone do not describe the full expense of handling a support request.

Track human corrections and escalations alongside API usage. A cheaper response is not necessarily cheaper support if it creates more follow-up work. Decide your acceptable quality, latency, and budget thresholds before selecting the winner.

#7 · Pilot with controls in place

Launch with a limited audience or a draft-only workflow where agents approve replies. Enforce authentication, permissions, and validation in application code before any account-changing action; model instructions alone are not an authorization system. Keep a clear path to human support.

If a multi-model relay suits your setup, create a Relay365 account and run your evaluation there. Its API base is https://api.api365relay.com; OpenAI clients add /v1. Keep credentials on your server and confirm the exact model identifiers before integration.

#8 · Revisit the decision regularly

Save your evaluation cases and rerun them when you change the model, prompt, retrieval setup, or tools. Add real failures as regression tests. The right model is the one that keeps meeting your support requirements in practice—not the one with the most persuasive launch announcement.