Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Recently we have been seeing the Merge Queue get stuck:
- https://github.com/apache/datafusion/issues/25088
We enquired with ASF Infrastructure on [INFRA-28374](https://issues.apache.org/jira/browse/INFRA-28374) and they told us the issue is that the ASF's GitHub Action queue is full which is why it is taking a long time to start jobs:
> We are aware of a long term usage pattern whereby our limited number of
> concurrent GitHub Actions jobs running on GitHub hosted runners is at 100%
> utilisation most of the day most of the time. Thus, from time to time we
> communicate to projects who utilise the most minutes to review their
> workflows for efficiency improvements.
>
> At the moment, datafusion is the top consumer of GitHub Action minutes over
> the last 7 days, and 3x the 10th in the list of minutes used.
Note the "datafusion" project as measured by ASF Infra includes all DataFusion subprojects (`datafusion`, `datafusion-comet`, `datafusion-ballista`, `datafusion-python`, etc.), so reducing usage may require a cross-repository effort.
Here is the [report of our current usage](https://infra-reports.apache.org/#ghactions&project=datafusion&hours=168&limit=15&group=name) (login required):
### History
This is the second time we have been contacted by ASF Infra about GHA usage.
The first was in **May 2026**, when ASF Infra identified DataFusion as a top-5 consumer: https://github.com/apache/datafusion/issues/22455. After some tweaks, by June 2026, our usage dropped below the policy thresholds and DataFusion fell out of the top 5.
### Useful resources
- ASF GitHub Actions policy (limits: 250,000 minutes / calendar week, 216,000 minutes / any 5-day window): https://infra.apache.org/github-actions-policy.html
- ASF usage report for the datafusion project (login required): https://infra-reports.apache.org/#ghactions&project=datafusion&hours=168&limit=15&group=name
- ASF Infra best-practices wiki page: https://cwiki.apache.org/confluence/spaces/INFRA/pages/430408443/GitHub+Actions+Recommended+Practices
- ASF Slack channel for sharing workflow optimisations across projects: `#project-workflow-optimisations`
### Related issues
- #22455 (previous round of this same problem)
- #13813
- #21048
- #25088 (CI takes hours to start — likely a symptom of pool saturation)
- #21241 (discussion of when extended tests should run)
- #25011 (example of a runaway CI job: extended tests hitting the 360 minute limit)
Contributor guide
Research direction
Start with the GitHub Actions workflows across the DataFusion subprojects listed in the issue, then compare them with the ASF usage report and recommended practices. Review related issues #22455, #21241, and #25011 for prior optimisations and costly jobs. Done means a documented reduction in GHA usage that helps prevent queue saturation across the project.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust
- Domain
- ci-cd, devops
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100