Improve test reliability
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 935
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 17
Description
tests are still failing too often. Since we don't have CI stats for github, I tried to crawl workflow results using a rudimentary [bash script](https://gist.github.com/mbien/6d5f82f6fbb6c67bfbb702bf91fe3c0c) calling into the github API.
The script scans workflow runs and filters merges to master which failed, then scans re-runs too and prints the failed steps. Rest is just `sort` and `uniq`.
```
14 "ide/project.ant" - done
13 "java.hints batch1"
11 "java.file.launcher" - done
10 "ide/css.editor"
7 "groovy/groovy.editor" - done
6 "webcommon/html.knockout"
3 "platform/o.n.bootstrap"
2 "java/maven"
2 "java.hints batch2"
1 "web.beans"
1 "platform/openide.nodes"
1 "java.editor"
```
note: this does not take the retry script into account which might be active on [some steps](https://github.com/apache/netbeans/blob/cbfd04788a960b293b2b16521629aabdafee1e84/.github/workflows/main.yml#L701)
here the results with the links pointing directly to the workflow run:
run: https://github.com/apache/netbeans/actions/runs/12936028241 (4 attempts)
- `groovy/groovy.editor`
- `groovy/groovy.editor`
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12934523662 (2 attempts)
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12933263724 (3 attempts)
- `ide/project.ant`
- `ide/project.ant`
- `java.editor`
- `java/maven`
run: https://github.com/apache/netbeans/actions/runs/12933078681 (4 attempts)
- `java.hints batch1`
- `java.hints batch2`
- `platform/o.n.bootstrap`
- `java.hints batch1`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12912321783 (3 attempts)
- `java.file.launcher`
- `java.hints batch1`
- `platform/o.n.bootstrap`
- `ide/css.editor`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12872488235 (3 attempts)
- `webcommon/html.knockout`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12869051716 (3 attempts)
- `java.hints batch1`
- `ide/css.editor`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12857668868 (2 attempts)
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12851818621 (3 attempts)
- `ide/css.editor`
- `webcommon/html.knockout`
run: https://github.com/apache/netbeans/actions/runs/12846158264 (2 attempts)
- `ide/project.ant`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12836525165 (2 attempts)
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12836146985 (2 attempts)
- `java/maven`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12826345964 (3 attempts)
- `ide/project.ant`
- `platform/o.n.bootstrap`
- `webcommon/html.knockout`
- `ide/project.ant`
run: https://github.com/apache/netbeans/actions/runs/12825920011 (3 attempts)
- `ide/project.ant`
- `groovy/groovy.editor`
- `ide/project.ant`
run: https://github.com/apache/netbeans/actions/runs/12825908774 (2 attempts)
- `ide/project.ant`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12795877604 (2 attempts)
- `ide/project.ant`
- `java.hints batch1`
- `java.file.launcher`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12794975127 (2 attempts)
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12791936642 (2 attempts)
- `ide/project.ant`
- `groovy/groovy.editor`
run: https://github.com/apache/netbeans/actions/runs/12765772367 (2 attempts)
- `ide/project.ant`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12764904231 (2 attempts)
- `ide/project.ant`
- `java.file.launcher`
- `platform/openide.nodes`
run: https://github.com/apache/netbeans/actions/runs/12762278637 (3 attempts)
- `java.file.launcher`
- `ide/css.editor`
- `ide/css.editor`
run: https://github.com/apache/netbeans/actions/runs/12762263024 (3 attempts)
- `java.hints batch1`
- `java.file.launcher`
- `java.file.launcher`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12749609504 (2 attempts)
- `java.hints batch1`
- `java.hints batch2`
- `java.file.launcher`
run: https://github.com/apache/netbeans/actions/runs/12722000825 (4 attempts)
- `ide/project.ant`
- `web.beans`
- `groovy/groovy.editor`
- `java.hints batch1`
- `ide/css.editor`
- `java.hints batch1`
- `java.hints batch1`
run: https://github.com/apache/netbeans/actions/runs/12721993365 (2 attempts)
- `ide/project.ant`
- `webcommon/html.knockout`
Contributor guide
Research direction
Start by reviewing the referenced .github/workflows/main.yml retry configuration around line 701 and the linked bash script that scans GitHub Actions workflow runs. Use the listed failed steps and workflow-run links to identify recurring failures; done means the affected test jobs are made reliably passing rather than requiring repeated runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, github-actions, java
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100