A journey towards automatic changelog/releasenotes

Background:

Logstash's changelog has always been maintained as a manual process. This is usually produced as part of the release process just prior to publishing. It's hard. It's complex. It's error prone. It takes a lot of energy.

Problem: It takes a lot of effort on the release engineer (usually @suyograo?). This is an error-prone, time-consuming process.

Solution: Automate it.

Implementation options (non-exhaustive):

There are two areas of concern:

1) Discovery of changes that were put into a given release
2) Communication of those changes to users.

We currently use labels on PRs to indicate what branches they were merged into. Labelling is an error-prone process but solving is out of scope for this ticket.

I talked with @suyograo. We are both happy with the current discovery mechanism (github search for labels), so what remains is how we communicate these changes.

Existing communication strategies:

Elasticsearch, as far as I know, uses PR/issue titles to communicate the changelog (which are gathered into the release notes). This seems like a good idea, but I feel it is not a good approach for Logstash. The reason is that I believe most PRs are communications between developers, and I believe the changelog should be communication on how a change impacts users. These two audiences are often quite independent and often, in my experience, requiring different language and detail.

---

Proposal:

In order to help us accurately and quickly produce a changelog for a given release, we should do the following:

* Add a special note to any PR/issue with language targeted at users -- what is the user impact of the change, how should a user prepare for the change, where to learn more, what to expect, etc.
* Write a script collect all of these special notes to produce the changelog for given release.
* Write a script that will remind us of any PR/issue that is missing a changelog note.

I imagine 3 states an issue/pr can have once it is marked resolved ("closed" on github):

1) Has a changelog entry.
2) Does not need a changelog entry because it has no user impact and is marked as such. Similar to rdoc's `:nodoc:` in idea.
3) Does not have a changelog entry at all.

For 1 and 2, these are accepted by the reminder script proposed above. For 3, this should strongly remind us that a change does not have an accompanying changelog entry.

Notification proposal:

If a change (PR/etc) does not have a changelog entry, then maybe we have this script automatically comment on the PR to request a changelog entry be written.

Summary:

* Embed changelog entry (user impact, etc) into the issue/pr description or in a comment on that issue/pr..
* Script to produce a changelog from this data for a given release
* Script to help enforce that we are writing changelog entries on issues
* All of this would be done through Github (changelog content on issues, discovery of changes in a given release, etc)

----

Next steps:

* See if there's any reusable prior art. Surely other projects may have ventured down this road?
* If nothing reusable is found, we need to define the syntax of what a changelog entry might look like in a github issue/pr description or comment.

Start with the issue's proposal for GitHub-based changelog notes, existing labels, and release discovery via GitHub search. Research reusable prior art, then define the changelog-entry syntax and the scripts' expected behavior; done means the release-generation and reminder workflows are specified well enough to implement.

Written by the indexing model from the issue text.