Duty Log: Keep These Internal Docs Chatbot Practices

The internal documentation chatbot has reached the duty desk when someone asks it a question during an incident and acts on the answer. That is the standard worth building for this year: answers that respect access rules, identify their sources, and make uncertainty visible before it becomes an operational mistake.

Keep the practices that make those answers dependable. Start with document ownership, retrieval boundaries, and a small set of realistic questions. Model selection belongs in that work, but it cannot repair a stale runbook or grant permission to read a restricted incident report.

Keep document ownership attached to every answer

Before indexing the wiki, establish which sources are authoritative. A service runbook, an approved policy, and an engineer’s scratch notes can describe the same system differently. Preserve each document’s owner, revision, approval status, and last review date as metadata.

Split documents along meaningful boundaries, keeping prerequisites and warnings attached to the procedures they qualify. Every retrieved passage should retain its parent document and section. A paragraph about restarting a worker becomes dangerous if retrieval drops the warning about draining its queue first.

• Assign an owner who can resolve conflicting instructions for each operational area.
• Track source revisions so changed and deleted documents trigger index updates.
• Show the source’s review date separately from the chatbot’s ingestion date.

Keep access checks inside the retrieval path

Authenticate the user before searching and enforce permissions against the passages supplied to the model. Hiding restricted citations after generation leaves the restricted content available to influence the answer. Search filters must reflect the user’s current access, including group membership changes and revoked permissions.

Apply the same boundary to caches, conversation history, and diagnostic logs. A cached answer can expose a restricted policy even when the next retrieval request is correctly filtered. Include authorization context in cache decisions, and invalidate affected entries when permissions change.

Keep citations close to claims and uncertainty explicit

Require citations for operational instructions and policy claims. A useful citation opens the relevant section and lets the reader check the supporting passage. Test whether the passage actually supports the claim; a plausible document title is insufficient evidence.

Define what happens when sources disagree or retrieval finds nothing useful. The chatbot should describe the gap, identify conflicting revisions where available, and direct the employee to the responsible owner. For questions involving production changes, make missing prerequisites explicit before presenting a procedure.

Treat retrieved text as untrusted input. A document can contain instructions that attempt to redirect the assistant or expose information. Keep system instructions separate from source material, and give any connected tools their own authorization checks. Retrieved prose must never grant execution privileges.

Keep a regression set from actual desk traffic

Collect representative questions from support tickets, onboarding requests, and incident reviews. Include exact service names, ambiguous abbreviations, outdated procedures, restricted documents, and questions with no documented answer. Remove sensitive details before putting examples into a shared evaluation set.

Evaluate retrieval and answer quality separately. First check whether the right authorized passages were found. Then check whether the response follows those passages, cites them accurately, and handles missing evidence appropriately. Run the set whenever you change chunking, prompts, embedding models, or generation models.

Track latency and cost alongside correctness. Measure the full request path, including retrieval and retries, because a quick model response can still sit behind a slow search service.

Keep model routing behind a measured interface

A common client interface makes model experiments easier to review. Relay365 fits teams that want pay-as-you-go access to multiple model families through an OpenAI SDK compatible interface. You could evaluate Gemini 2.5 Flash and GPT-6 Astra against the same internal question set, then choose routes from observed answer quality, latency, and cost.

Before routing internal documents through any provider or relay, confirm its data handling and retention terms against your organization’s requirements. Keep credentials on the server, set request budgets, and record which model and configuration produced each evaluated answer.

If that access model fits your deployment, create a Relay365 account and begin with a controlled evaluation using approved documents. Keep the release gate concrete: employees receive authorized evidence, citations support the instructions, and unanswered questions reach someone who owns the documentation.