apache / apache/netbeans

Improve test reliability

Open
#8,183 7 comments 0 reactions 0 assignees View on GitHub
CI kind:meta tests
Dominant language
Java
Stars
3.1k
Forks
935
Avg merge
2d 3h
Merged PRs (30d)
17

Description

tests are still failing too often. Since we don't have CI stats for github, I tried to crawl workflow results using a rudimentary [bash script](https://gist.github.com/mbien/6d5f82f6fbb6c67bfbb702bf91fe3c0c) calling into the github API.

The script scans workflow runs and filters merges to master which failed, then scans re-runs too and prints the failed steps. Rest is just `sort` and `uniq`.

```
14 "ide/project.ant" - done
13 "java.hints batch1"
11 "java.file.launcher" - done
10 "ide/css.editor"
7 "groovy/groovy.editor" - done
6 "webcommon/html.knockout"
3 "platform/o.n.bootstrap"
2 "java/maven"
2 "java.hints batch2"
1 "web.beans"
1 "platform/openide.nodes"
1 "java.editor"
```
note: this does not take the retry script into account which might be active on [some steps](https://github.com/apache/netbeans/blob/cbfd04788a960b293b2b16521629aabdafee1e84/.github/workflows/main.yml#L701)

here the results with the links pointing directly to the workflow run:

run: https://github.com/apache/netbeans/actions/runs/12936028241 (4 attempts)

- `groovy/groovy.editor`
- `groovy/groovy.editor`
- `groovy/groovy.editor`

run: https://github.com/apache/netbeans/actions/runs/12934523662 (2 attempts)

- `java.file.launcher`

run: https://github.com/apache/netbeans/actions/runs/12933263724 (3 attempts)

- `ide/project.ant`
- `ide/project.ant`
- `java.editor`
- `java/maven`

run: https://github.com/apache/netbeans/actions/runs/12933078681 (4 attempts)

- `java.hints batch1`
- `java.hints batch2`
- `platform/o.n.bootstrap`
- `java.hints batch1`
- `java.hints batch1`

run: https://github.com/apache/netbeans/actions/runs/12912321783 (3 attempts)

- `java.file.launcher`
- `java.hints batch1`
- `platform/o.n.bootstrap`
- `ide/css.editor`
- `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12872488235 (3 attempts)

- `webcommon/html.knockout`
- `webcommon/html.knockout`

run: https://github.com/apache/netbeans/actions/runs/12869051716 (3 attempts)

- `java.hints batch1`
- `ide/css.editor`
- `webcommon/html.knockout`

run: https://github.com/apache/netbeans/actions/runs/12857668868 (2 attempts)

- `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12851818621 (3 attempts)

- `ide/css.editor`
- `webcommon/html.knockout`

run: https://github.com/apache/netbeans/actions/runs/12846158264 (2 attempts)

- `ide/project.ant`
- `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12836525165 (2 attempts)

- `java.file.launcher`

run: https://github.com/apache/netbeans/actions/runs/12836146985 (2 attempts)

- `java/maven`
- `java.file.launcher`

run: https://github.com/apache/netbeans/actions/runs/12826345964 (3 attempts)

- `ide/project.ant`
- `platform/o.n.bootstrap`
- `webcommon/html.knockout`
- `ide/project.ant`

run: https://github.com/apache/netbeans/actions/runs/12825920011 (3 attempts)

- `ide/project.ant`
- `groovy/groovy.editor`
- `ide/project.ant`

run: https://github.com/apache/netbeans/actions/runs/12825908774 (2 attempts)

- `ide/project.ant`
- `java.file.launcher`

run: https://github.com/apache/netbeans/actions/runs/12795877604 (2 attempts)

- `ide/project.ant`
- `java.hints batch1`
- `java.file.launcher`
- `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12794975127 (2 attempts)

- `groovy/groovy.editor`

run: https://github.com/apache/netbeans/actions/runs/12791936642 (2 attempts)

- `ide/project.ant`
- `groovy/groovy.editor`

run: https://github.com/apache/netbeans/actions/runs/12765772367 (2 attempts)

- `ide/project.ant`
- `java.hints batch1`

run: https://github.com/apache/netbeans/actions/runs/12764904231 (2 attempts)

- `ide/project.ant`
- `java.file.launcher`
- `platform/openide.nodes`

run: https://github.com/apache/netbeans/actions/runs/12762278637 (3 attempts)

- `java.file.launcher`
- `ide/css.editor`
- `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12762263024 (3 attempts)

- `java.hints batch1`
- `java.file.launcher`
- `java.file.launcher`
- `java.hints batch1`

run: https://github.com/apache/netbeans/actions/runs/12749609504 (2 attempts)

- `java.hints batch1`
- `java.hints batch2`
- `java.file.launcher`

run: https://github.com/apache/netbeans/actions/runs/12722000825 (4 attempts)

- `ide/project.ant`
- `web.beans`
- `groovy/groovy.editor`
- `java.hints batch1`
- `ide/css.editor`
- `java.hints batch1`
- `java.hints batch1`

run: https://github.com/apache/netbeans/actions/runs/12721993365 (2 attempts)

- `ide/project.ant`
- `webcommon/html.knockout`

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the referenced .github/workflows/main.yml retry configuration around line 701 and the linked bash script that scans GitHub Actions workflow runs. Use the listed failed steps and workflow-run links to identify recurring failures; done means the affected test jobs are made reliably passing rather than requiring repeated runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, github-actions, java
Domain
ci-cd, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.