Duty Log: Support Bot Model Selection After Failures

The support queue does not need a model leaderboard. It needs correct answers, completed actions, and clean handoffs. Treat model selection as an incident review: reproduce the failure, isolate the cause, and change the model only when the evidence points there.

The bot invents refund rules. Should we upgrade?

First, inspect what the bot received. Replay the failed conversation with the applicable policy, effective date, and account facts explicitly included. Ask it to identify supporting evidence or request clarification instead of filling gaps. This is a diagnostic test, not a promise that better context fixes every failure.

Compare candidates on that same replay. Grade policy correctness separately from tone, and inspect the actual outcome rather than accepting a convincing answer as success. Agent evaluations should distinguish what the assistant says from what happened in the system. ([anthropic.com](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents))

The answer sounds right, but the refund never happens.

Inspect tool calls before replacing the model. Check whether the refund tool was available, its purpose was clear, required arguments were supplied, and its response reached the conversation. Tool descriptions, schemas, and returned context are part of the interface the model must understand. ([anthropic.com](https://www.anthropic.com/engineering/writing-tools-for-agents))

For your selection test, require the expected backend state, not merely a sentence announcing completion. Separately test denied requests, expired sessions, and interrupted calls. Keep authorization checks in application code, and require confirmation where your workflow demands it.

Customers abandon the chat while the bot thinks.

Measure the whole path before blaming generation speed. Record model time, retrieval time, tool time, and retries separately. Track both the first useful response and completed resolution; a quick greeting is not the same as quick help. Tool evaluations can capture runtime, call counts, token consumption, and errors alongside accuracy. ([anthropic.com](https://www.anthropic.com/engineering/writing-tools-for-agents))

Then test routing. Send routine policy questions through one path and ambiguous technical cases through another. Routing can separate customer-service categories into specialized workflows, but it depends on accurate classification. Include misrouted tickets in the test rather than assuming the router always chooses correctly. ([anthropic.com](https://www.anthropic.com/engineering/building-effective-agents))

The cheaper model creates more repeat contacts.

Define your purchasing metric as cost per correctly resolved case, not price per token. For your own workload, include retries, escalations, and human cleanup in that calculation. Set acceptance criteria before comparing candidates so attractive pricing cannot quietly lower the quality bar.

A practical trial could include Claude Haiku 4.5 and Gemini 2.5 Flash as candidates, without assigning either a winner in advance. Use the same sanitized tickets, policy snapshots, tools, and scoring rules. Run repeated trials because model behavior can vary between attempts. ([anthropic.com](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents))

The demo passes, but production conversations unravel.

Replace polished opening questions with complete conversations. Include corrections, missing order numbers, conflicting requests, and explicit requests for a human. Evaluate both transcripts and final outcomes; multi-turn agents can make mistakes that propagate across subsequent actions. ([anthropic.com](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents))

• Require the bot to preserve corrected account details throughout the conversation.
• Check that an escalation includes the issue, attempted steps, and unresolved questions.
• Test whether the bot stops when a required permission or tool is unavailable.
• Keep a separate regression set that was not used to tune prompts.

The next shift needs a repeatable model trial.

Relay365 fits when the immediate job is comparing model families through a pay-as-you-go relay desk. Use it as an access layer for your evaluation, not as evidence that different models have identical behavior. Before sending customer records, review applicable data-handling terms and test the features your workflow requires.

Create a Relay365 account when you are ready to run the same sanitized support suite across candidates. Keep the selection decision attached to measured outcomes rather than familiarity with a model name.

The shift handoff should record the selected model, prompt revision, tested failure cases, routing rules, and rollback trigger. Leave the next engineer a reproducible decision, not a note that the bot seemed better.