siderolabs / siderolabs/docs

Reference architecture: run Enterprise Image Factory self-hosted and air-gapped

Open
#770 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
MDX
Stars
11
Forks
76
Avg merge
3d 5h
Merged PRs (30d)
26

Description

Parent issue for documenting a self-hosted, air-gapped Enterprise Image Factory. This is the deliverable behind the "Running the factory self-hosted and air-gapped, documented" line on the fall launch readiness table, deferred from the September 15 Talos Enterprise Linux launch and tracked here as a fast follow.

The September entitlement includes self-hosting rights for Enterprise Image Factory and the Discovery Service, and on-prem EIF is announced at launch. So a customer can buy this and read nothing about how to run it, which is the gap. A prospect asking for FIPS images on-premises was already told it is doable but undocumented.

Scope

One reference architecture. Say this is how you run a self-hosted air-gapped factory, and support that. Not a matrix of permutations. That scoping was set at the August 18 readiness review and should not be reopened while writing.

The outcome to aim at: an on-prem customer can generate their own Talos Enterprise Linux images and trust them the way they would trust ours. The second half is where the unresolved design questions are.

Do not start from scratch

Most of the material exists. Reuse rather than rewrite.

Published on the docs site:

  • Run Image Factory On-Prem is substantial and current. It covers the internal registry with certificates or insecure, a fully air-gapped path, the image cache signing key, cache storage, connected versus custom-registry factory modes, seeding, cosign signing, self-signed CA variants, and the factory config for each.
  • Run Omni Air-gapped is its companion and carries the certificate generation steps.

In siderolabs/image-factory/docs/, developer-facing and unpublished: air-gapped.md with hack/copy-artifacts.sh and the artifacts.core.registry / namespace config for registries that prefix the upstream path, plus authentication.md, configuration.md, sources.md, cache.md, api.md and attestations/. There is also a Helm chart (chart.md, deploy/helm) distributed via OCI with Chainsaw e2e tests, which the published guide does not mention because it only shows docker run.

Mechanics to reuse rather than reinvent: talosctl image talos-bundle <release> lists every image needed to seed a factory (Talos 1.12 and later); talosctl image cache-create / cache-cert-gen / cache-serve is the sanctioned way to carry images across an air gap before pushing with crane, and it exists because Talos releases pin container digests that have to match in the internal registry; the factory verifies container signatures on pull, cosign v2 only, via containerSignature.publicKeyFile; and cache.signingKeyPath and authentication.downloadTokenKeyPath must both be set explicitly for any multi-replica deployment, because keys generated at startup break behind a load balancer.

What the enterprise page has to add over the community one

  1. The enterprise build. The published guide runs ghcr.io/siderolabs/image-factory, the open-source image. enterprise/ is BUSL 1.1 and is not built by default, so the page needs whatever a customer actually pulls or builds. Not treated as a risk: the source is in the public repo and Omni ships the same way. It just needs the real image reference.
  2. Seeding from the hosted factory. The hosted enterprise factory acts as an authenticated registry source for enterprise Talos images, gating on entitlement, and an on-prem factory is configured to point at it. Air-gapped deployments put a registry mirror in between. Document which config keys, which credential, and the mirror variant.
  3. Signatures and provenance. See the open questions below.
  4. Placement. The existing page lives under omni/self-hosted/ and ends by handing off to Omni, so a Talos Enterprise Linux customer without Omni never finds it. Per the documentation IA rule, the enterprise section owns a complete page wherever the enterprise product changes the workflow, so this gets its own version rather than a link.

The last mile stays the customer's problem. There is no Sidero-supplied offline transport, so the page documents how to configure the source and how to mirror, and stops there.

Open questions, to answer while writing rather than before

  • Whose key signs the output. Signing and attestation live in the enterprise build (enterprise/assetsignature, enterprise/installerattestation). If a customer-run factory signs with their key, output is verifiable internally but not attributable to Sidero, which is the reason a compliance buyer bought the product. If it signs with ours, we have put a signing key on customer infrastructure. There may be a third answer where our signatures on the seeded artifacts pass through and the factory re-attests nothing.
  • Whether re-signing is needed at all. The community guide has the customer generate a cosign key and re-sign every image. If the enterprise path points at our factory as the registry source, the artifacts already carry our signatures and containerSignature.publicKeyFile can be our public key. Re-signing may apply only to a true offline mirror. Confirm before copying that section across.
  • SBOM, VEX and scan reports. enterprise/spdx, enterprise/vex and enterprise/scanner are in the enterprise build, so a self-hosted factory presumably produces its own. Are they equivalent to the hosted factory's, and does the customer need anything an air gap denies them, such as a scanner database?
  • Which credential the on-prem factory presents when pulling from the hosted factory as a registry source. Likely a machine-scoped token.
  • Whether an image built on-premises carries the same FIPS claim. A compliance customer air-gapping the factory will ask, and it connects to the FIPS wording work.
  • docker run or the Helm chart. The chart exists but is not on the docs site. For an enterprise on-prem deployment the chart is probably the honest answer.

Out of scope

Authentication choices on a self-hosted factory are the customer's: htpasswd, or no auth, or their own tenant. Auth0 specifics, the Organizations workaround, and the hosted machine-credential gap are hosted-factory concerns and do not belong on the reference architecture page. The exception is bringing your own Auth0 tenant, which is a real self-hosting task and is tracked separately in #720 as a child of this issue.

Testing

Exercising the reference architecture is the test. There is no separate air-gap test to define. The factory's own test runs already use a private registry mirror, so they are close to air-gapped already.

Origin

Action item from the August 18 readiness review: create a task ticket to document the deployment process for Enterprise Image Factory in air-gapped and on-prem environments. Filing it now, with the research from the intervening weeks folded in.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the published Run Image Factory On-Prem and Run Omni Air-gapped pages, then compare them with image-factory/docs/air-gapped.md, authentication.md, configuration.md, sources.md, cache.md, api.md, and attestations/. Exercise the reference architecture using the factory's private-registry test setup; done means an enterprise customer can deploy, seed or mirror, and generate verifiable images with the enterprise image, credentials, signatures, and deployment path documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm
Domain
devops, documentation, infrastructure
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.