fluent / fluent/fluent-bit

workflows: nightly build updates and regular release cadence

Open
#4,875 6 comments 0 reactions 1 assignee Claimed by @patrick-stephens View on GitHub
ci enhancement
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 16h
Merged PRs (30d)
58

Description

Once 1.9 release is out we want to move to a regular nightly build that produces a floating `unstable` tag.
We also want to support building from a branch (e.g. `master` or `1.9`) without a tag and then creating a new tag (e.g. for `1.9.1`) at the end of the process once we're happy.

We also need to distinguish the version we build from (i.e. the commit/branch) from the version we're creating (i.e. the tag/release for a new official version).
Currently the process is all assuming the current workflow of building tag X to make release X.

We want to move to commit rather than tag-based release process:
- Regularly build (nightly) latest commit on `master`
- Create a new/updated `unstable` Github release for this with all artefacts - derisked already.
- Update testing to use this unstable release URL for artefacts and containers.
- On successful testing move to `staging` as "ready to release" - plus this can then trigger performance ad resilience tests.
- The current release workflow will then take from staging (currently implemented) and create the tag (new).

The last step can also be automated to have a regular release cadence a well.
The goal is to always have something ready to release, i.e. the last green staging build.

There are some additional changes to get in here, e.g. update the containers to build from a commit rather than a tarball of the last tagged version.
This is already underway though for the multi-arch containers as well.

Currently the staging approach uses an S3 bucket and will overwrite this with the nightly build.
This update will simplify this so only "good" builds are in staging plus reduce S3 transfer costs.

# Overview

```mermaid
graph TD
direction LR
Commit[Commit] -->|Cron - nightly| Unstable(Unstable)
Unstable --> Testing{Testing}
Testing -->|Passes| Staging(Staging)
Testing -->|Fails| Commit
Staging -->|Cron - weekly| Release(Release)
Release -->|Update next version| Commit
```

As you can see, we should always have something to release from staging - if testing fails it does not go into staging. We can then just push whatever is in staging out to release.

# Tasks

- unstable release per branch: https://github.com/fluent/fluent-bit/pull/5117
- 1.8 builds
- staging to fix: https://github.com/fluent/fluent-bit/runs/6005595200?check_suite_focus=true
- test [ ] - need to adjust the smoke test targets
- release [ ]
- pull from unstable release and test - update current staging test which uses bucket
- if successful, push to staging per branch - likely need to rebuild to remove the `nightly` information and then retest
- take staging and release (per branch)

- add SHA of Git commit to version output #5006
- remove special characters in debug output #5002
- add automated check to verify version and commit for release, prevent #4970
- investigate how to promote the same binary rather than rebuilding for staging, issue is with the nightly setting
- support cmake versioning from the CI, so the build can generate the new version possibly to check in after release so people can update after if needs be - release pushes version, CMakeLists gets updated and checked in for main branch then
- Add the pre-built workflow [Create Release](https://github.com/MylesBorins/node-osc/blob/main/.github/workflows/create-release.yml)
- Add in the workflow [Bump Version](https://github.com/MylesBorins/node-osc/blob/main/.github/workflows/bump-version.yml)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.