elastic / elastic/apm-agent-java

Improve GH actions release process

Open
#3,186 7 comments 0 reactions 0 assignees View on GitHub
agent-java
Dominant language
Java
Stars
594
Forks
338
Avg merge
1d 13h
Merged PRs (30d)
25

Description

Version `1.39.0` was the first attempt of releasing via GH actions. It uncovered a few problems.

1st attempt:
* There was a bug in the job which calls build-kite to perform the release, which meant that `mvn deploy was not triggered`
* The release preparation step already had committed the version bumps and the new tag.
* Resolution:
* Fix the GH-action via a PR
* Revert the version bump commits via a PR
* Manual deletion of the tag (required admin permissions!)
* Retry in 2nd attempt.

2nd attempt:
* The `Deploy to maven central` "failed", however the actual artifact was actually published to maven central correctly. We think that the final "publish" of the release returned a strange response / timed out, but actually went through.
* As a result, all of the follow-up tasks were aborted due to "failed" maven central publish step
* Resolution:
* We added boolean inputs to skip the `prepare release` and `Deploy to maven central` steps
* This made everything work except for the AWS-lambda upload, as it depends on the uploaded lambda-zip from the buildkite build. Subsequently, also the creation of the GH release failed.
* Those had to be done manually (required admin permissions and AWS credentials).

The GH actions release workflow was designed with atomicity of the individual steps in mind, so that in theory a failure could be mitigate by just using the `Rerun failed Jobs` feature. Unfortunately, this does not take the case where a job fails but silently succeeds (e.g. our maven deploy) into account, which is where things went wrong.
In addition, the `Rerun failed jobs` does not work correctly if there is a bug in the workflow or a related script. To my knowledge, the `Rerun failed jobs` would always use the same revision, therefore no fixes to the scripts can be applied.

We'll most likely need to split the process up into individual workflows, which ideally are idempotent.
We especially should adjust the lambda-task to not depend on a previous uploaded build artifact, but to use the apm-agent from maven central.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.