vectordotdev / vectordotdev/vector

Migrate packages.timber.io artifacts to dd-cose-releases and retire obsolete content

Open
#26,104 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: delivery domain: releasing
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

Goal

Migrate Vector release artifacts from packages.timber.io to the Datadog-owned dd-cose-releases bucket, move first-party consumers to the new endpoint, then retire obsolete content and eventually decommission the old hosting.

Current state

  • Provision dd-cose-releases through Cloud Inventory with public object reads, encryption, logging, and prefix-scoped publisher/operator access (DataDog/cloud-inventory#68539).
  • Publish future stable, nightly, and custom builds to the vector/ prefix using GitHub OIDC (#26154).
  • Verify the custom-build workflow can publish to vector/custom/ and that the objects are publicly downloadable.
  • Migrate 106 immutable stable-version prefixes through 0.57.0: 2,952 objects totaling 108,325,104,776 bytes. All 1,636 non-empty source artifacts matched the destination by size; two broken zero-byte aliases whose redirect targets did not exist were intentionally omitted.
  • Stop publishing stable .X aliases to COSE (#26163).
  • Decide not to expose stable vector/latest/ or vector/latest.txt in COSE (#26168, merged into #26079).
  • Switch the website, installer, documentation, and Homebrew release tooling to exact-version COSE URLs and publish the deprecation notice (#26079).
  • Dual-publish the next stable release to both endpoints and verify its checksums and public downloads.
  • After that release, stop all new Vector writes to packages.timber.io.
  • Delete legacy nightlies older than seven days while preserving the rolling seven-day window until the old hosting is retired.
  • Complete consumer and retention checks for old custom builds, legacy Timber products, obsolete Helm content, and the remaining frozen stable objects before deletion.

Decisions

  • dd-cose-releases is the COSE-owned shared release-artifact bucket. Vector owns the vector/ prefix; Quickwit can later use an independently permissioned quickwit/ prefix.
  • The supported public artifact root is https://dd-cose-releases.s3.amazonaws.com/vector/. Exact object URLs are public; bucket and prefix listing is intentionally disabled.
  • Stable COSE artifacts use only immutable vector/<exact-version>/ prefixes. COSE does not provide stable latest, latest.txt, minor .X, or major .X aliases.
  • GitHub Releases continue receiving stable assets as an additional mirror and release-discovery surface, but first-party direct downloads use COSE.
  • The next stable release is dual-published to COSE and packages.timber.io. Later releases publish only to COSE.
  • Existing packages.timber.io URLs remain available during the deprecation window. After writes stop, retained objects remain frozen until their announced deletion/decommission date.
  • Historical nightly and custom objects are not migrated. Future nightlies publish under vector/nightly/<date>/ and vector/nightly/latest/; future custom builds publish under vector/custom/.
  • Old-bucket nightlies older than seven days are deleted. Old custom builds are deleted later after retention and consumer checks.
  • Helm chart packages remain GitHub Release assets; helm.vector.dev remains the repository-index endpoint.
  • Legacy Timber product artifacts (agent/, cli/, and cloudwatch-logs-lambda-function/) and obsolete Helm prefixes are not migrated and are deleted only after their own consumer and retention checks.

Remaining plan

  1. Merge #26079 and publish the user-facing deprecation announcement in Vector 0.58.0.
  2. Dual-publish and verify the next stable release at both endpoints.
  3. Stop all subsequent Vector writes to packages.timber.io while leaving retained objects readable and frozen.
  4. Add automated seven-day retention for old-bucket nightlies.
  5. Announce the final retention/decommission date after consumer checks, then delete the approved legacy content and decommission the old hosting.
  6. Add Quickwit later with a separate publisher identity and quickwit/ prefix; neither project receives write access to the other's prefix.

Acceptance criteria

  • Exact-version Vector artifacts are publicly downloadable from dd-cose-releases/vector/ and first-party stable-download consumers use that endpoint.
  • The next stable Vector release is published and checksum-verified at both the old and new endpoints.
  • Future nightly and custom builds publish only to COSE; stable builds publish only to exact-version COSE prefixes after the one-release transition.
  • After the transition release, no new Vector artifacts are written to packages.timber.io.
  • GitHub Releases continue receiving stable release assets.
  • The deprecation entry identifies the replacement URL, migration guidance, old-URL availability, and retirement boundaries.

Pending decision

  • Final retention/deletion date for frozen packages.timber.io stable objects and the remaining approved legacy content.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the remaining plan and PR #26079, then inspect the release-publishing workflows introduced by #26154 and the existing stable, nightly, and custom consumers. Verify the next stable release at both endpoints, switch first-party downloads to exact-version COSE URLs, and confirm retention, deprecation, and no-new-write acceptance criteria before retiring approved legacy content.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, github-actions
Domain
cloud, devops, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.