Duty Log: Prompt Release Held Until These Checks Pass

A prompt release needs a decision, not a reassuring demo. Before shipping, the duty engineer should be able to identify the tested prompt, explain its known failure modes, and restore the previous version. Use the checklist below as release gates: every required gate must pass, and unresolved blockers mean no-go.

Freeze the candidate before collecting evidence

Record the prompt version, model identifier, generation settings, tool definitions, retrieval configuration, and application build. Test the complete message sequence, including system instructions and inserted context, rather than an isolated paragraph copied into a chat window.

• The candidate is stored as a versioned artifact that another engineer can reproduce.
• The intended behavior and release-blocking failures are written down before the test run.

No-go: The tested configuration differs from the configuration scheduled for deployment. A last-minute wording change creates a new candidate and needs another relevant test run.

Run a small suite that includes uncomfortable inputs

Build a compact test set from representative tasks, previous incidents, and foreseeable misuse. Include ordinary requests, ambiguous instructions, missing information, long context, conflicting constraints, and requests outside the product’s scope. Use synthetic or appropriately sanitized data.

• Each test has an expected outcome or an explicit scoring rubric, not merely a reference answer that looks plausible.
• The suite includes cases where the correct behavior is to ask a question, decline a request, or state uncertainty.
• Variable outputs are tested repeatedly, with the repetition count chosen and recorded before evaluation.

No-go: A release-blocking case fails, or reviewers cannot agree on what passing means. Fix the acceptance criteria before interpreting the results.

Verify the output contract in the real application

A good-looking answer can still break a parser or trigger the wrong action. Run responses through the same validation and tool-handling code used in production. Separate syntactic validity from semantic correctness: valid JSON can still contain an unauthorized operation.

• Structured outputs pass the actual schema, including required fields, allowed values, and size limits.
• Tool arguments are validated, and consequential actions retain application-side authorization and any required confirmation.
• Refusals, empty responses, truncated responses, and tool errors produce a controlled application state.

No-go: The application depends on the prompt alone to enforce permissions or prevent destructive actions.

Challenge instruction boundaries and data handling

Place hostile instructions inside user messages, retrieved documents, and tool results. Check whether the assistant treats those materials as task data rather than as authority to override its operating instructions. Test attempted disclosure of sensitive context without putting real secrets into the fixture.

• Injection attempts do not produce unauthorized tool use or disclose protected data.
• Logs and evaluation artifacts follow the application’s data-handling requirements.

No-go: A boundary failure remains unexplained or unfixed. Do not average a security failure away with successful routine answers.

Check the target model and operating budget

Evaluate the exact deployment target. If you support both Claude Haiku 4.5 and Gemini 2.5 Flash, keep separate results; passing on one does not establish readiness on the other. Measure end-to-end latency, token usage, retries, and timeout behavior against limits your team has explicitly approved.

Relay365 can fit this workflow when you need pay-as-you-go access across model families through a shared integration. Its API base is https://api.api365relay.com; OpenAI clients add /v1. A shared endpoint simplifies wiring, but does not establish identical model behavior, tool support, or output reliability.

• The candidate meets the documented quality and operating limits on every supported deployment target.
• Fallback behavior is tested rather than assumed to preserve the primary model’s results.

Record the decision and keep rollback ready

Attach the test report, unresolved limitations, release owner, and rollback instructions to the release record. Use staged exposure where practical, and define observable conditions that stop the rollout.

If multi-model testing fits your workflow, create a Relay365 account and run the same versioned suite against your intended targets.

Go means the required gates passed, accepted limitations are documented, and someone owns monitoring and rollback. Anything less is a hold, not a conditional production experiment.