Duty Log: AI Ticket Triage Practices We’re Keeping

This year, the useful role for AI in support triage is still narrow: read the incoming report, suggest a category, identify missing information, and propose the next queue. Keep the practices that make those suggestions inspectable. Retire the ones that turn a plausible summary into an unreviewed operational decision.

The desk should finish each shift knowing which tickets moved automatically, which needed review, and which failed processing. A tidy dashboard is not enough if nobody can reconstruct why a customer landed in the wrong queue.

Keep the routing contract smaller than the ticket

Define the output before choosing a model. A useful triage record contains an allowed category, a proposed queue, an urgency recommendation, supporting excerpts, missing fields, and a review flag. Validate those fields in application code. Reject unknown categories instead of silently translating them into a convenient default.

Separate customer impact from customer language. A frustrated message is not automatically a critical incident, and a polite report can describe a widespread outage. Give the model explicit severity criteria grounded in your incident policy, then check important claims against trusted account and service data.

• Allow the system to return an unknown category when the evidence does not fit.
• Require supporting excerpts for urgency recommendations without requesting hidden reasoning.
• Version the routing taxonomy and prompt so that later audits can reproduce the decision context.

Keep customer text outside the control plane

A ticket is untrusted input, even when it arrives from a familiar customer. Instructions embedded in a message or attachment must not change the triage policy, reveal internal information, or authorize tool calls. Delimit ticket content clearly, but do not treat formatting alone as a security boundary.

Enforce permissions outside the model. The triage worker should not be able to issue refunds, reset credentials, or alter account access. Redact unnecessary personal information before sending requests, restrict retrieved context to relevant records, and apply retention rules to prompts, outputs, and logs.

When a ticket involves suspected account compromise, legal threats, or sensitive personal information, route it through a defined review path. Let policy determine that path rather than asking the model to improvise one.

Keep evaluation close to actual queue failures

Build a review set from appropriately handled historical tickets, including ambiguous reports, multilingual messages, duplicate incidents, and misrouted cases. Preserve a held-out set so prompt edits do not merely memorize the examples used during development.

Measure routing accuracy by category, missed urgent cases, unnecessary escalations, and agent corrections. Also track processing cost and latency. A single aggregate accuracy score can hide a weak category that creates substantial downstream work.

Compare candidate models on the same cases and output contract. For example, Gemini 2.5 Flash and Claude Haiku 4.5 can be evaluation candidates without assuming either is suitable before testing. Treat model-reported confidence as an uncalibrated signal, not permission to automate.

Keep humans in charge of consequential moves

Start in shadow mode: generate recommendations without changing queues. Next, enable only the low-risk routes that meet your own acceptance criteria. Keep escalation, account-sensitive decisions, and uncertain cases available for human review.

Make correction cheap. Agents should be able to change a queue, record a reason, and inspect the supporting excerpts without opening another investigation. Feed those corrections into reviewed evaluation data rather than automatically treating every override as ground truth.

Plan for timeouts, malformed responses, and unavailable providers. Preserve the original ticket, use bounded retries, and fall back to ordinary intake. Log failures distinctly from successful classifications.

Keep provider flexibility behind a tested adapter

Relay365 is worth considering when the desk needs pay-as-you-go access across model families through one integration surface. It works with OpenAI SDK clients using https://api.api365relay.com/v1 as the base URL. Keep provider-specific behavior behind an adapter, and test structured output, errors, and fallback behavior before changing production routing.

If that fits your evaluation workflow, create a Relay365 account and run a privacy-reviewed ticket sample before enabling automation. A shared endpoint simplifies integration; it does not replace model evaluation or data-handling review.

The practice to carry forward is straightforward: automate the reversible sorting work, preserve evidence, and leave consequential judgment with accountable people.