gitpod-io / gitpod-io/gitpod

[image builder] count failures and differentiate between system and user

Open
#15,573 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component: image-builder meta: never-stale team: workspace
Dominant language
TypeScript
Stars
13.8k
Forks
1.4k
Avg merge
17h 58m
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe

Image builds fail, but, we don't know how many of those failures are related to the system, versus user input.

Describe the behaviour you'd like

  • when the system fails, count as a system failure, either as a label on an existing metric, and/or one or more new metrics. The intent is to grant us the ability to tell how many system failures fail image builds overall.
  • use the build output (which is on stderr) to identify user errors
    • create a set of parsing rules for known user errors. E.g. a RUN command in the Dockerfile exiting with an error. This would contain a log with >>> RUN:
      image

    • we can incrementally add to this list of parsing rules whenever we identify new user errors. For this, it would be useful for the rule list to be specified in config, so it becomes easy to update.

  • an image build could fail if we cannot authenticate with a private registry to pull the private base image, this should be considered a user failure (introduced in https://github.com/gitpod-io/gitpod/pull/8550)
  • there are likely other paths that require instrumentation, ☝️ are just initial thoughts

Additional context

Related: https://github.com/gitpod-io/gitpod/issues/15572

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with components/image-builder-mk3/pkg/orchestrator/metrics.go, especially the existing metrics referenced in the issue, then trace image-build failure paths and the private-registry authentication path introduced by PR 8550. Done means system and user failures are distinguishable in metrics, known stderr user errors are covered by configurable parsing rules, and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Domain
infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.