How to Write a Runbook for an AI-Powered Service

An AI service runbook should help someone act safely when the system behaves unexpectedly. It is not an architecture essay or a collection of dashboard links. It is a practical sequence that explains what to inspect, what to change, and how to confirm recovery. Follow these steps to write a first version that a newcomer can actually use.

Define the AI service and its boundaries

1. Start with a short description of what the service does, who depends on it, and which failures matter most. Describe the complete request path, including your application, retrieval system, relay or model provider, and any tools the model can invoke. Name the service owner and the person authorized to disable risky functionality.

Distinguish an unavailable service from an available service producing unacceptable answers. Both need operational attention, but they require different checks and recovery actions.

Prepare access before an incident begins

2. List the dashboards, logs, configuration controls, and deployment tools an operator needs. Explain how to request access without placing passwords, API keys, or customer content in the runbook. Include where credentials are managed and how an authorized operator can revoke or rotate them.

• Identify the dashboard that shows request failures, latency, and traffic volume.
• Explain how to locate a request using its correlation identifier.
• Record where operators can inspect the active model, prompt version, and deployment version.

Turn alerts into clear starting instructions

3. Give each alert a plain-language explanation and a first action. An elevated error rate should direct the operator to inspect error categories. A latency alert should prompt a check of application processing time, retrieval time, and upstream response time.

Set thresholds using observed service behavior and business requirements rather than borrowing arbitrary numbers. Explain whether latency measures the first streamed token or the complete response. Document which conditions require immediate escalation, especially suspected data exposure or unauthorized tool execution.

Separate application failures from model behavior

4. Write a diagnostic sequence that narrows the problem before anyone changes production. First establish when the issue began and which users, routes, or request types are affected. Then compare that window with deployments, configuration changes, and upstream errors.

For example, if a route uses Gemini 2.5 Flash, record its configured identifier and inspect representative failed requests with sensitive content removed. Check authentication, request validation, rate limits, timeouts, retrieval results, and output parsing separately. A successful HTTP response does not establish that the answer was correct or safe.

Document safe recovery actions and stopping conditions

5. For each likely failure, describe one recovery action, its prerequisites, and its reversal procedure. Useful actions include rolling back a prompt, disabling a tool, limiting traffic, or restoring a previous deployment. State who can authorize each change and when the operator must stop and escalate.

If your service uses Relay365, its pay-as-you-go relay desk can fit into this section as an upstream access layer across model families. Its compatibility with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients can also help teams use familiar tooling. Document the configuration your service actually uses rather than assuming every client has identical behavior.

Do not make model switching an automatic cure. Test alternate routes for output format, tool behavior, safety requirements, latency, and cost. Bound retries, and prevent duplicate side effects when a timed-out request may already have performed an action.

Verify recovery with representative requests

6. Define the evidence required before declaring recovery. Check technical signals and run approved test requests that exercise retrieval, structured output, and tool use where applicable. Avoid using real customer data unnecessarily.

Record the observation period your team requires, who confirms recovery, and how customer-facing updates are approved. If you are evaluating Relay365 for this access layer, create an account and validate your intended integration before adding it to production recovery procedures.

Rehearse and maintain the runbook

7. Ask a teammate unfamiliar with the service to follow the instructions in a safe environment. Replace ambiguous directions with exact commands or navigation steps. Assign an owner and record the last review date. After incidents and significant changes, update the runbook so the next operator starts with tested instructions rather than outdated assumptions.