Duty note: keep AI log triage evidence-bound

The useful question for this year’s operations playbook is not whether AI can read logs. It is whether the next person on duty can verify its findings without repeating the investigation. Keep the practices that shorten the path from noisy events to a testable hypothesis. Drop the ones that produce confident summaries without a trail back to the evidence.

Start each investigation with a bounded evidence packet

Do not hand a model an unfiltered incident window and ask what happened. Define the affected service, environment, time range, timezone, observed symptom, and any recent change. Include the query that selected the events and state whether the results were sampled, truncated, or filtered.

A useful first request is narrow: group recurring errors, identify changes around the first observed failure, and cite the events supporting each finding. Ask the model to separate direct observations from hypotheses and missing evidence. This makes an incomplete investigation visible instead of letting fluent prose conceal it.

Require citations that survive the shift handoff

Every material claim should point to a retrievable record: a timestamp plus event identifier, a trace ID, or another stable reference supported by your logging system. Preserve the original evidence outside the generated answer. A copied message fragment alone may be ambiguous when thousands of requests produce identical errors.

• Verify quoted events against the original records before using them in an incident update.
• Check numerical claims with deterministic queries rather than accepting counts calculated from a model’s excerpt.
• Keep event time, ingestion time, and deployment time distinct when reconstructing a sequence.
• Record alternative explanations when the available logs cannot establish causation.

A timeout appearing after a deployment is a lead, not proof that the deployment caused it. Confirm the proposed mechanism using metrics, traces, configuration changes, or a controlled test.

Treat log contents as untrusted input

Logs regularly contain user-controlled strings, request payloads, and copied error responses. Those strings can include instructions aimed at the assistant. Treat them as evidence to inspect, never as authority to change the task, retrieve credentials, or execute commands.

Redact secrets and unnecessary personal data before sending records to a model. Use an allowlist of fields where practical, and review retention, access controls, and data-handling terms for every service in the request path. A relay adds another boundary to assess; it does not remove the need for that review.

Keep query execution separate from remediation

Let AI propose a query or explain a stack trace before allowing it to operate tools. Begin with read-only access, bounded time windows, result limits, and explicit permissions. Inspect generated queries for expensive scans and accidental scope expansion.

Require human approval for restarts, rollbacks, configuration changes, and destructive actions. The assistant should describe the proposed action, supporting evidence, expected effect, and verification step. A plausible explanation is not an authorization mechanism.

Evaluate models on incidents your team understands

Build a small, sanitized replay set from resolved incidents, including confusing cases and incidents with insufficient evidence. Compare outputs against the known record. Track unsupported claims, citation accuracy, useful next queries, latency, and total investigation cost. Measure reviewer effort as well as response speed.

For example, compare Gemini 2.5 Flash and Claude Haiku 4.5 on the same evidence packets rather than assuming either is suitable from its name. Keep prompts and tool permissions consistent, and rerun the checks when models or routing settings change.

Use a relay where multi-model evaluation earns its place

Relay365 is worth considering when the duty desk needs pay-as-you-go access to multiple model families through a shared integration. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Evaluate its data path, operational dependencies, and costs alongside the models themselves. If that setup fits your workflow, create a Relay365 account and start with sanitized incident replays.

The handoff standard stays simple: preserve the query, evidence references, uncertainty, and next verification step. Keep AI in the investigation loop because its work is inspectable, not because its summary sounds finished.