actions / actions/runner-container-hooks

Surface pod Events (FailedScheduling, etc.) on waitForPodPhases timeout

Open Beginner friendly
#366 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
153
Forks
112
Avg merge
6m
Merged PRs (30d)
1

Description

Problem

When a workflow pod fails to reach Running for a reason that lives in K8s Events rather than on the pod object itself, the hook surfaces only:

##[error]Error: pod failed to come online with error: <generic timeout>
##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.

The most common cases:

Event What the user actually needs to see
FailedScheduling 0/14 nodes are available: 13 Too many pods, 1 node was unschedulable.
FailedScheduling Insufficient cpu / Insufficient memory
FailedScheduling node(s) had untolerated taint …
Failed (kubelet) Error: ErrImagePullBackOff predecessors

The ephemeral workflow pod is typically pruned before an operator can kubectl describe it, so the diagnostic is lost.

Scope vs other PRs

  • #336 handles containerStatuses[].state.waiting.{reason,message} on the pod object — covers ImagePullBackOff, missing tags, etc. Doesn't read Events.
  • #341 (merged) fixes the {} empty-message problem in 4 throw sites. Doesn't read Events.
  • #364 (open) fixes the circular-JSON crash. Doesn't read Events.

This issue is the missing third piece: read pod Events when the pod object alone doesn't explain the failure.

Proposed fix (~15 LOC)

In k8s/index.ts waitForPodPhases, in the catch / timeout path, before throwing, fetch the most recent Warning events for the pod and append them to the error message — best-effort, swallow any API failure:

let extra = ''
try {
  const events = await k8sApi.listNamespacedEvent({
    namespace: namespace(),
    fieldSelector: `involvedObject.name=\${podName},type=Warning`
  })
  const warnings = (events.items ?? [])
    .sort((a, b) => +new Date(b.lastTimestamp ?? b.eventTime!) -
                    +new Date(a.lastTimestamp ?? a.eventTime!))
    .slice(0, 3)
    .map(e => \`[\${e.reason}] \${e.message}\`)
  if (warnings.length) extra = \`; events: \${warnings.join('; ')}\`
} catch { /* diagnostic best-effort */ }
throw new Error(
  \`Pod \${podName} is unhealthy with phase status \${phase}: \${formatError(error)}\${extra}\`
)

Unit test: mock listNamespacedEvent to return a FailedScheduling: 0/3 nodes are available: 3 Too many pods. warning; assert that substring shows up in the thrown error.

Happy to open the PR

If the proposal looks right, I can open a PR mirroring the structure of #336 + #364 — small, tightly scoped, with a unit test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in k8s/index.ts at waitForPodPhases and inspect its timeout or error path. Add a unit test that mocks listNamespacedEvent with a FailedScheduling warning, then verify the warning text appears in the thrown error and that event lookup failures remain best-effort.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, typescript
Domain
devops, infrastructure
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.