An On-Call Checklist for Production AI Features

Your first AI incident does not need a clever response. It needs a repeatable sequence: establish impact, contain damage, identify the failing layer, and verify recovery. Keep this checklist beside your dashboards, deployment history, and escalation contacts. Follow it in order unless an active security issue requires immediate containment.

Establish the incident owner and user impact

1. Acknowledge the alert and name the person coordinating the response. Open an incident channel and record the start time, affected feature, and known symptoms. Separate investigation from stakeholder updates when enough responders are available. Prioritize reducing user impact over proving the root cause; mitigation can precede a complete explanation. ([sre.google](https://sre.google/workbook/incident-response/?utm_source=openai))

• Identify whether users cannot complete tasks, receive incorrect answers, or experience delayed responses.
• Record the affected customer groups, deployment versions, and request routes.
• Set the next update time and identify the escalation contact.

Trace one failed request across the AI path

2. Choose one representative failure and follow it through your application, retrieval service, relay if present, model endpoint, and downstream tools. Capture timestamps, application trace IDs, and available upstream request IDs. OpenAI recommends logging request IDs in production to support troubleshooting; preserve identifiers at each boundary rather than assuming one identifier follows the entire request. ([developers.openai.com](https://developers.openai.com/api/reference/overview?utm_source=openai))

Check the actual destination, credentials, model identifier, timeout configuration, and recent configuration changes. Use a sanitized reproduction rather than copying customer content into a shared incident channel.

Separate transport failures from answer failures

3. Classify the symptom before changing settings. Treat authentication errors, throttling, timeouts, malformed output, and incorrect answers as separate investigation paths. Compare a failing request with a known-good test using the same configuration. For answer problems, inspect retrieved evidence, prompt changes, output validation, and tool results before blaming the model.

Keep reproduction tests read-only. Do not replay a request that can send messages, change records, or charge a customer until you have checked whether its original side effects already occurred.

Stop retries from amplifying the incident

4. Inspect both application retries and SDK retries. For retryable rate-limit failures, use bounded exponential backoff with jitter, honor applicable server retry instructions, and keep retries within the user-facing deadline. Do not repeatedly retry billing or quota failures that require intervention. Account for SDK behavior so nested retry loops do not multiply requests. ([developers.openai.com](https://developers.openai.com/api/docs/guides/rate-limits?utm_source=openai))

Set explicit concurrency and queue limits. If the deadline has expired, return a clear failure or an approved degraded experience instead of continuing invisible work.

Contain unsafe actions and unexpected spending

5. Disable affected write-capable tools when action safety is uncertain. Require confirmation for consequential operations, restrict tool permissions, and verify deduplication before replaying work. Review token usage, request volume, and tool execution counts against your own normal baseline.

Use preconfigured per-user usage limits or hard caps to constrain excessive consumption; OpenAI documents these as protections against high-volume misuse. Choose thresholds from your workload and business tolerance, not an arbitrary universal number. ([developers.openai.com](https://developers.openai.com/api/docs/guides/rate-limits?utm_source=openai))

Activate only a tested fallback route

6. Prefer a known-good rollback, disabled feature, or pretested fallback over an improvised migration. Google’s incident-response guidance emphasizes having generic mitigations such as rollback ready before an emergency. ([sre.google](https://sre.google/workbook/incident-response/?utm_source=openai))

If your application uses several model families, consider Relay365, a pay-as-you-go relay desk, as part of that routing setup. Before an incident, test each intended route for streaming, structured output, tool behavior, and data-handling requirements. Treat compatibility as something to verify for your feature, not permission to switch blindly.

Verify recovery before restoring full traffic

7. Run representative checks for successful completion, valid output, expected latency, and safe tool execution. Include the original failure case and a normal customer journey. Restore traffic gradually, watching errors, queue depth, usage, and answer quality. Keep the mitigation available until the recovered path remains healthy under your defined acceptance criteria.

Leave the next responder a usable runbook

8. Record the timeline, mitigation, contributing changes, and follow-up owners. Add the sanitized reproduction to your regression tests and document exact rollback and escalation steps. If a multi-model relay fits your architecture, create a Relay365 account and validate its routes in staging before placing them in the production runbook.