A Practical Incident Review for Model API Outages

An incident review should explain what users experienced, how responders restored service, and what needs to change. Keep it blameless: examine the conditions behind decisions rather than judging individuals with hindsight. Google’s postmortem guidance emphasizes impact, investigation, and actionable follow-up. ([sre.google](https://sre.google/workbook/postmortem-culture/?utm_source=openai))

Use the numbered walkthrough below as a reusable document. Replace each bracketed field with evidence, and write “unknown” when an answer is not yet established. The examples are hypothetical, not reports of actual outages.

Establish the model API incident record

1. Give the review a clear identity before reconstructing events. Assign one document owner, distinguish the incident status from the investigation status, and record which application workflow failed.

• Record the incident identifier as [ID], the review owner as [name], and the current investigation status as [status].
• Identify the affected application, requested model identifier, client version, endpoint path, and routing configuration.
• Summarize the symptom in one sentence, such as “Requests to Gemini 2.5 Flash timed out during document summarization.”

Describe the outage impact with measured evidence

2. Document the affected services, incident window, and user-visible consequences. Make the impact assessment complete enough to support follow-up decisions, consistent with Google’s postmortem checklist. ([sre.google](https://sre.google/workbook/postmortem-culture/?utm_source=openai))

• Write the first confirmed failure and last confirmed failure as timestamps with an explicit timezone.
• Record affected requests, affected users where measurable, and the denominator behind any reported failure rate.
• Separate failed requests from slow responses, interrupted streams, and outputs that users could not use.

Reconstruct the API outage timeline

3. Build a chronological record from logs, alerts, deployment records, and responder notes. Include detection, investigation, mitigation, and recovery milestones rather than relying on memory alone. Detailed incident timelines are part of Google’s suggested postmortem tooling. ([sre.google](https://sre.google/workbook/postmortem-culture/?utm_source=openai))

Use one sentence per event: “[UTC timestamp] — [observation or action] — [supporting evidence].” Distinguish when an event happened from when someone discovered it. Include unsuccessful mitigations and explain what responders knew at that moment.

Preserve diagnostic evidence without exposing secrets

4. Collect request identifiers, response codes, sanitized error messages, latency observations, and configuration changes. Define redaction rules before sharing telemetry; OpenTelemetry notes that sensitivity depends on your context and requires deliberate handling. ([opentelemetry.io](https://opentelemetry.io/pl/docs/security/handling-sensitive-data/?utm_source=openai))

Exclude API keys, authorization headers, and unnecessary prompt or response content from the review. Keep restricted evidence in an approved location, and reference its internal identifier. For streaming requests, record whether failure occurred before the first token or after partial output.

Separate the failure trigger from contributing conditions

5. Write three short statements: what changed, how that change produced the observed failure, and which conditions increased the impact. Treat each explanation as a hypothesis until evidence supports it.

Investigate the application, client configuration, network path, relay, and upstream service separately. Do not call a timeout an upstream outage without corroboration. If the cause remains unknown, document what has been ruled out and assign the next diagnostic task.

Record mitigation and demonstrate API recovery

6. Describe every mitigation, its execution time, observed result, and rollback condition. Keep immediate restoration work distinct from longer-term corrective actions, as Google’s incident guidance does. ([static.googleusercontent.com](https://static.googleusercontent.com/media/sre.google/en//static/pdf/IncidentManagementGuide.pdf?utm_source=openai))

For recovery, record successful checks of the affected workflow, including streaming or tool use if applicable. Specify the observation window and acceptance criteria. If you switched models or routes, test output suitability before declaring the workaround successful.

Evaluate a relay without assuming automatic resilience

7. If your response involved routing changes, document who controlled each route and what evidence identified the failing layer. Consider Relay365’s pay-as-you-go relay desk when evaluating access to multiple model families. Assess it against your application’s compatibility, data-handling, observability, and cost requirements.

If that approach fits your architecture, create a Relay365 account and test a non-production workflow. Do not assume that relay access provides automatic failover or independent failure domains; verify the intended behavior before depending on it.

Assign outage prevention work and close the review

8. Finish with tracked actions rather than general promises. Google recommends prioritizing postmortem work and following action items through completion. ([sre.google](https://sre.google/workbook/postmortem-culture/?utm_source=openai))

• Give each action one owner, a priority, a due date, and a measurable completion test.
• Replace “improve monitoring” with a specific change and a test showing that it detects the documented failure.
• Ask responders to verify the timeline, preserve unresolved questions, and schedule a check on outstanding actions.