Duty Desk Note: AI Model Checks Before Go-Live

A model that gives an impressive demo is not automatically ready for production. Production means real users, imperfect inputs, bills, outages, and consequences when an answer is wrong. Before approving a launch, use this checklist to decide whether the model fits the job and whether your team can operate it safely.

Define the job before choosing the model

Start with one specific task, such as drafting support replies or extracting fields from invoices. Avoid requirements like “make the application smarter.” You need a clear description of the input, the expected output, and what happens when the model cannot complete the task.

• Write down which decisions the model may make and which require a person.
• Define an acceptable result using examples that your team can review consistently.
• Decide whether a simpler rule, search query, or existing workflow would solve the problem more predictably.

Test ordinary work and uncomfortable edge cases

Build a small evaluation set from representative work, removing sensitive information where necessary. Include incomplete requests, unusual formatting, ambiguous instructions, and questions with no supported answer. Keep some cases separate from prompt development so you can check whether improvements generalize.

You might compare Gemini 2.5 Flash and Claude Haiku 4.5 on the same cases. Treat their names as candidates, not evidence of suitability. Measure correctness, useful refusals, output formatting, and the amount of human editing needed. Repeat important tests because answers can vary.

Trace where production data will travel

Map the full path from your application through any relay service to the model provider. Each organization in that path may have different rules for logging, retention, access, and processing location. Do not assume that an API connection makes confidential data safe to send.

• Confirm applicable data-use terms, retention settings, deletion options, and contractual requirements.
• Remove unnecessary personal information and keep API keys out of prompts and client-side code.
• Check what your own logs capture, including prompts, responses, tool results, and error messages.

Keep generated text away from unchecked authority

A model can produce convincing but incorrect text. It can also encounter malicious instructions hidden in documents, websites, or messages. This is commonly called prompt injection: untrusted content tries to influence what the system does.

Treat model output as untrusted input. Validate structured responses in application code. Give connected tools only the permissions they need, and require approval for consequential actions such as sending money or deleting records. A prompt saying “be careful” is not an access-control system.

Budget for a completed task, not one request

Model APIs often charge by tokens, which are pieces of text processed or generated. The bill for a useful outcome can also include retries, long conversation histories, tool calls, and human review. Test costs with realistic workloads rather than a short demonstration prompt.

• Measure response time and cost per successfully completed task.
• Set limits for output length, retries, spending, and concurrent requests.
• Check documented rate limits and test behavior under expected traffic, including slower responses and rejected requests.

Verify the access layer without assuming equivalence

If you need to evaluate several model families, Relay365 is a pay-as-you-go relay desk worth considering. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. Its API base is https://api.api365relay.com; OpenAI clients add /v1.

A shared access point can simplify integration, but it does not establish identical model behavior or feature support. Verify the exact model identifier, streaming, tool calling, error responses, billing, and data handling for your intended route. Near deployment, create a Relay365 account if that access pattern fits your requirements, then run your evaluation through the actual production path.

Leave the next shift a rollback plan

Release to limited traffic first. Record the model identifier, prompt version, evaluation results, and responsible owner. Monitor failures and user corrections without collecting unnecessary sensitive data. Define when to stop requests, switch to a tested fallback, or hand work to a person. Approve broader rollout only when someone can explain both the evidence for launch and the procedure for undoing it.