Shift Note: Checks Before the AI Feature Goes Live

An AI feature can fail while its servers still look healthy. Requests may return successfully but contain unusable answers, take too long, or trigger expensive retries. On-call work means spotting those failures, limiting their impact, and leaving a clear record for the next shift. Start with this checklist before you carry the pager.

Before the shift, locate the controls that stop damage

Know the path from a user action to the model and back. Include your application, any relay service, the model provider, retrieval systems, and tools the model can call. A failure anywhere along that path can look like a model problem.

• Confirm who owns the feature, who can change its configuration, and who handles security or privacy incidents.
• Find the switches that disable generation, block tool actions, limit traffic, and restore the last known working configuration.
• Check where credentials live and how to revoke or rotate them without exposing them in chat or incident notes.
• Write down which user journeys require AI and which can continue with search, templates, or manual review.

At the first alert, measure the user-visible failure

A successful HTTP response only means a request completed at that layer. It does not prove the answer was useful. Watch both technical signals and the behavior users actually experience.

• Track failed requests, rate-limit responses, timeouts, response latency, and incomplete streams separately.
• Monitor invalid structured output, missing citations where required, empty answers, and failed tool calls.
• Compare current token usage and spending with normal traffic, and investigate sudden changes in usage per request.
• Run a small set of safe test requests with known expectations, including one that checks whether required output fields are present.

Set alerts around your own service targets and baseline behavior. There is no universal latency or error threshold that fits every AI feature. A background summarizer and a live support assistant need different treatment.

During triage, preserve evidence without copying secrets

Capture the time, request identifier, configured model, prompt version, deployment version, and relevant error details. Separate failures affecting every request from failures limited to one model, tenant, region, or input type.

Use redacted examples when investigating answer quality. Prompts and responses may contain personal information, credentials, or confidential documents. Keep access restricted and follow your retention policy rather than pasting raw conversations into a shared incident channel.

When service degrades, contain it before changing models

• Apply request deadlines and bounded retries, with delays between attempts, so a slow dependency does not create a growing queue.
• Check whether retrying could repeat a tool action, such as sending a message or creating an order, before allowing automatic retries.
• Reduce concurrency or disable nonessential AI work when rate limits, queues, or spending rise unexpectedly.
• Switch to a tested fallback or a clear unavailable message when you cannot safely serve the normal experience.

A fallback model is a behavior change, not just a routing change. If your backup uses Gemini 2.5 Flash, test its formatting, tool behavior, latency, and answer quality against your actual workload before an incident. Keep permission checks and approval gates in application code, regardless of the model.

For a shared relay, keep dependency checks separate

Relay365 is a pay-as-you-go relay desk for multiple model families. It can fit teams that want a shared access point, and it works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Still, distinguish application errors, relay errors, and upstream model errors in your logs. Shared access does not make different models interchangeable or remove the need to test recovery paths.

If that setup matches your workflow, create a Relay365 account and validate a small, non-sensitive workload before routing production traffic through it.

Before handover, record the recovery and remaining risk

Verify recovery with real user-facing checks, not just a quiet dashboard. Record what changed, which mitigations remain active, who owns follow-up work, and when temporary limits should be reviewed. Leave the next engineer a short timeline and a working recovery path—not a mystery wrapped in a green status indicator.