02:00 Shift Note: Keep AI Jobs Bounded and Recoverable
The overnight queue should leave the morning team with finished work or a clear exception report, not a mystery. For this year’s scheduled AI jobs, the practices worth keeping are operational rather than theatrical: narrow permissions, durable state, bounded spending, and a recovery path that does not depend on someone remembering the original prompt.
Give the overnight job a small, testable contract
Start with an assignment whose completion can be checked without asking another model for its opinion. Summarizing new support tickets into a staging table is a reasonable unattended task. Resolving every urgent customer issue is not a sufficiently bounded one.
Define the input window, permitted tools, output schema, and destination. Separate producing a recommendation from executing it. A job can prepare a pull request or draft a response overnight while leaving merge or delivery approval to a person.
- The job should reject missing inputs rather than invent substitute records.
- The output should pass schema validation and task-specific checks before it becomes visible downstream.
- Instructions found inside tickets, documents, or repository files should remain untrusted data, not authority to expand tool access.
Make the schedule survive missed and overlapping runs
A cron expression is only the trigger. Specify the time zone, daylight-saving behavior, and policy for missed runs. Decide whether a delayed invocation should process the original window, skip it, or combine it with a later batch. Use explicit window boundaries rather than interpreting “yesterday” inside the prompt.
Assign each run a durable identity based on the job and input window. Use a lease or lock to prevent overlapping workers from processing the same batch. Store checkpoints outside the worker process so a restart does not erase progress.
Keep an action ledger as well. Model requests and external writes are different operations: repeating a summary request may cost money, while repeating a customer notification can create an incident. Use destination-supported idempotency keys where available, and reconcile uncertain write outcomes before trying again.
Stop retries before they become the night’s workload
Set request timeouts, a whole-run deadline, concurrency limits, and a retry budget. Retry transient failures with backoff and jitter. Do not keep retrying invalid credentials, malformed requests, or repeatedly invalid outputs without changing the conditions.
Distinguish a transport failure from a poor answer. A successful HTTP response does not establish that the task succeeded. Conversely, a timeout after an external write does not establish that nothing happened. Record these states separately so recovery follows evidence rather than guesswork.
Cap input size, output tokens, tool calls, and total iterations. Track estimated spend during the run and reconcile it with available usage records afterward. Estimates are useful controls, but they are not necessarily final charges. When a limit is reached, save progress and stop cleanly.
Choose model access without hiding operational differences
Use a fixed evaluation set before changing the model assigned to a scheduled job. Claude Haiku 4.5 could be a candidate for a bounded extraction task, but selection should follow measured correctness on your inputs. Any fallback needs the same validation; API compatibility does not mean identical behavior.
Relay365 is worth considering when a team needs pay-as-you-go access across several model families through one relay desk. It works with Claude Code, Codex, Gemini CLI, and OpenAI SDK clients. That can simplify connection configuration, but your scheduler still needs its own deadlines, permission boundaries, and output checks. Confirm data-handling requirements before routing production content through any intermediary.
Leave a handoff the morning operator can act on
Log the run identity, configuration version, model identifier, input counts, validation failures, usage, and final state. Keep secrets and unnecessary sensitive content out of logs. Alert immediately only when someone can usefully intervene; route ordinary exceptions into the morning queue with a clear next action.
If consolidated model access fits your workflow, create a Relay365 account and test one low-risk scheduled job before expanding the queue.
The useful morning handoff is short: what completed, what stopped, what changed externally, and where a human decision is required. Keep that standard this year. An unattended job earns its place by being inspectable and recoverable, not merely by running while everyone sleeps.
Further desk bulletins
| Date | Bulletin title | Filed under | Language |
|---|---|---|---|
| 2026-10-28 | Shift Note: One Python App, Several Model Providers | Field notes from the desk | EN |
| 2026-10-28 | Duty log: one handoff prompt, three different answers | Field notes from the desk | EN |
| 2026-10-28 | Shift Note: Temperature Settings Before the Next Run | Field notes from the desk | EN |
| 2026-10-28 | Shift Note: Keep System Prompts Small and Testable | Field notes from the desk | EN |
| 2026-10-28 | Shift note: Factual answers need an evidence check | Field notes from the desk | EN |
| 2026-10-28 | Shift Note: Teach the Output With Three Examples | Field notes from the desk | EN |
Curious how it runs on your own code? Sign up →