actions / actions/runner-container-hooks

Pod template `imagePullSecrets` silently overwrites `container.credentials` — should union the lists

Open Beginner friendly
#447 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
153
Forks
112
Avg merge
6m
Merged PRs (30d)
1

Description

Summary

In the Kubernetes hook, when a job specifies container.credentials and the pod
template (ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE) also defines spec.imagePullSecrets,
the template's value completely overwrites the image-pull secret generated from
container.credentials. The user-supplied job credentials are silently discarded — no
warning, no error. The $job container is then pulled using only the template's secret,
which fails with ImagePullBackOff if that secret has no auth entry for the job image's
registry.

imagePullSecrets is additive in Kubernetes (the kubelet unions all entries and matches
per registry host), so these two secrets should be merged (union + dedup by name), the
same way volumes already is — not overwritten.

Environment

  • actions/runner-container-hooks (k8s hook), main
  • ARC gha-runner-scale-set, containerMode: kubernetes / kubernetes-novolume
  • The workflow pod runs an ARC-injected fs-init init container using the runner image
    (on registry host A), while the $job container image is on a different registry
    host B provided via container.credentials.

Repro

  1. Set a hook template with a pod-level imagePullSecrets (e.g. to authenticate the
    fs-init runner image on a private mirror, registry host A):

    spec:
      imagePullSecrets:
        - name: infra-pull-secret   # covers registry host A (runner/fs-init image)
    
  2. Run a job whose container image is on a different private registry (host B), supplying
    credentials inline:

    container:
      image: registryB.example.com/toolbox:tag
      credentials:
        username: ${{ secrets.REG_B_USER }}
        password: ${{ secrets.REG_B_TOKEN }}
    
  3. Observe the created workflow pod: spec.imagePullSecrets contains only
    infra-pull-secret. The secret generated from container.credentials is gone, and the
    $job image pull fails with ImagePullBackOff (no auth entry for registry host B).

Root cause

createJobPod sets the container.credentials secret first, then merges the pod-template
extension:

// packages/k8s/src/k8s/index.ts (createJobPod)
if (registry) {
  const secret = await createDockerSecret(registry)
  const secretReference = new k8s.V1LocalObjectReference()
  secretReference.name = secret.metadata.name
  appPod.spec.imagePullSecrets = [secretReference]     // (1) job creds set here
}
...
if (extension?.spec) {
  mergePodSpecWithOptions(appPod.spec, extension.spec) // (2) template merged here
}

mergePodSpecWithOptions only merges containers and volumes; every other field
(including imagePullSecrets) hits the generic overwrite branch:

// packages/k8s/src/k8s/utils.ts
export function mergePodSpecWithOptions(base, from): void {
  for (const [key, value] of Object.entries(from)) {
    if (key === 'containers') {
      base.containers.push(...)
    } else if (key === 'volumes' && value) {
      base.volumes = mergeLists(base.volumes, value as k8s.V1Volume[])
    } else {
      base[key] = value            // imagePullSecrets clobbered here
    }
  }
}

So step (2) overwrites the array set in step (1), and container.credentials is lost.

Impact

  • Any setup where the runner/fs-init image and the $job image live on different
    private registries cannot authenticate both on the same pod. You are forced to bake a
    single multi-host secret into the hook template covering every registry any workflow
    might ever use, and container.credentials becomes dead config.
  • The failure is silent: valid, user-provided container.credentials are dropped with no
    log or warning.

Proposed fix

Treat imagePullSecrets like volumes in mergePodSpecWithOptions — union and dedup by
name rather than overwrite:

} else if (key === 'imagePullSecrets' && value) {
  base.imagePullSecrets = mergeLists(
    base.imagePullSecrets,
    value as k8s.V1LocalObjectReference[]
  )
}

Because createJobPod sets the container.credentials secret before the merge, the result
becomes [jobCredSecret, ...templateSecrets]: the kubelet uses the job creds for the
$job image and the template creds for the fs-init / infra image, matching per host.

Why this is safe

The union is a superset — every secret the template declared is still present. The only
behavioral change is that container.credentials is no longer discarded. Kubernetes already
treats imagePullSecrets as additive and dedups host matches, so extra entries are
harmless.

Alternatives considered

  • Multi-host secret in the hook template (current workaround): requires enumerating
    every registry up front and makes container.credentials inert — unworkable when jobs
    may use arbitrary registries.
  • ServiceAccount imagePullSecrets: the ServiceAccount admission controller only
    injects when the pod has no imagePullSecrets; since the hook already sets one, the SA
    entry is ignored — so this cannot supply the second secret.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mergePodSpecWithOptions in packages/k8s/src/k8s/utils.ts and then inspect createJobPod in packages/k8s/src/k8s/index.ts to confirm the merge order. Verify that generated pods retain the credential secret and template imagePullSecrets, with duplicate names removed; existing Kubernetes hook tests should cover the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, kubernetes, typescript
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.