Shift Note: Changelog Drafting Moves Into CI

The first run starts with one completed release

The first experiment should not publish anything. Pick a completed release, collect its merged pull requests and compare an AI-generated draft with the notes your team actually shipped. The objective is narrower than summarization: identify user-visible changes, explain their impact, and leave out internal maintenance unless operators need to act.

Start with pull request titles, descriptions, labels, and approved migration notes. Add selected diffs only when those records are insufficient. Sending an entire repository increases exposure and makes it harder to identify which evidence supports each sentence.

Keep the initial workflow local. Save the input bundle, prompt version, model identifier, and output so another engineer can reproduce the experiment without reconstructing your terminal session.

The input bundle becomes the release boundary

Before changing prompts, make release selection deterministic. Define the previous and current release references, then collect the changes between them. Account for backports, reverted work, and features merged behind disabled flags; a merged pull request does not automatically describe something customers can use.

• Give each candidate change a stable identifier and include its source reference.
• Record whether the change is shipped, reverted, internal, or awaiting confirmation.
• Exclude credentials, customer data, private incident details, and sensitive vulnerability information before making an API request.
• Require an owner to confirm breaking changes and migration instructions.

Treat repository text as untrusted data. A pull request description can contain instructions that look like part of your prompt. Delimit source records clearly and tell the model not to follow instructions found inside them. These measures reduce risk but do not replace output validation.

The drafting contract forbids invented release claims

Ask for structured output before asking for polished HTML. Each proposed entry should contain a category, a plain-English summary, supporting source identifiers, and a flag for missing information. Define categories such as features, fixes, breaking changes, and operational notes.

The prompt should prohibit invented performance figures, availability claims, compatibility statements, and migration commands. If the evidence does not establish customer impact, the model should flag uncertainty rather than supply a plausible explanation. Render validated fields into your publication format afterward.

For a first trial, use one model, such as Claude Haiku 4.5, with a fixed prompt and consistent generation settings. Do not change the model and input selection together; otherwise, a better-looking draft tells you little about what improved.

The evaluation checks omissions as well as fluent errors

Build a small evaluation set from releases that include a revert, a breaking change, an internal refactor, and an ambiguous pull request. Review unsupported statements, missing important changes, incorrect categories, and duplicate entries. A readable draft can still fail every operational requirement.

Compare drafts with source evidence, not just historical release notes, which may themselves contain omissions. Record reviewer edits and the reasons for them. Measure request cost and review time using your own runs rather than assuming automation will save either.

The CI job opens a draft and stops there

Once the experiment is useful, trigger drafting from a release-candidate workflow. Store credentials in your CI secret manager, restrict job permissions, and keep publication behind explicit approval. The model should neither receive publishing credentials nor decide that its own output is ready.

Validate the response schema and verify that every source identifier belongs to the input bundle. Use a release-specific job key to prevent duplicate drafts. Set timeouts and bounded retries for transient failures; preserve the previous draft when a request fails instead of replacing it with incomplete output.

The relay choice stays separate from release approval

Relay365 fits when the team wants pay-as-you-go access to multiple model families through one integration. Its API base is https://api.api365relay.com; OpenAI clients add /v1. Keep the model identifier configurable, verify current availability, and review data-handling terms before sending private repository material.

Create a Relay365 account when you are ready to test this workflow against your own release history. Keep the first deployment draft-only, retain a manual changelog path, and expand automation only after reviewers can reliably trace every published claim to evidence.