[Bug][GitHub] In-progress workflow runs are skipped and not collected after completion
- Dominant language
- Go
- Stars
- 3.1k
- Forks
- 808
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 49
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues.
### What happened
Hi DevLake team,
DevLake is a great platform, and it has already helped us solve several issues with our engineering data and reporting. We really appreciate all the work and effort from the DevLake team and community. Thank you for your continued support and for making such a valuable platform available to the community.
We are using Apache DevLake v1.0.2 with the GitHub plugin to collect GitHub Actions workflow runs.
### Environment
- DevLake version: v1.0.2
- Plugin: GitHub
- Deployment: AWS ECS
- Docker image: Official Apache DevLake image
- Sync mode: Incremental sync
- Workflow Run sync frequency: Every 2 hours (`0 */2 * * *`)
Our production GitHub Actions workflows can sometimes run for 3–4 hours or longer. Because of this, we configured the GitHub Workflow Runs sync policy to run every 2 hours so that completed workflow runs can be collected regularly.
### Problem
During the "Collect Workflow Runs" task, we see log messages such as:
Skipping run{id: 3372***, number: 1158***} with status in_progress
Skipping run{id: 3372***, number: 303***} with status in_progress
The workflow runs are therefore skipped while they are still `in_progress`.
Later, the GitHub Actions workflow finishes and GitHub changes the workflow run status to `completed`.
However, during subsequent incremental syncs, the previously skipped workflow run does not appear to be collected or updated after it becomes `completed`.
This means that a long-running workflow can be missed by the normal incremental sync if it is still `in_progress` when the collector checks it.
### Expected behavior
We would expect DevLake to:
1. Detect the workflow run while it is `in_progress`.
2. Retain/track the workflow run as unfinished.
3. Re-check the workflow run during subsequent incremental syncs.
4. Once GitHub reports the run as `completed`, update the existing workflow run record with the final status and information.
### Current behavior
Currently, the workflow run is skipped while it is `in_progress`.
If the workflow finishes after that collection cycle, the subsequent incremental sync does not appear to revisit the previously skipped run.
### Investigation so far
We also checked whether this behavior has been addressed in newer DevLake versions.
Based on our investigation, the same completed-only behavior appears to exist in v1.0.3 and v1.0.4-alpha. We also understand that the GitHub workflow collector was rewritten in newer development versions to address GitHub `/actions/runs` pagination limits, but it still appears to keep only runs with `status == "completed"`.
Therefore, upgrading the official image may improve GitHub collection reliability for large repositories, but it does not appear to address this specific long-running workflow scenario.
We also could not find an existing DevLake issue or PR specifically covering:
"Store in_progress GitHub Actions workflow runs and update them when they become completed."
The related pagination work appears to address a different problem.
### Questions
Could you please confirm:
1. Is this expected behavior in DevLake v1.0.2?
2. Is there any configuration that allows DevLake to re-check previously skipped `in_progress` GitHub workflow runs during subsequent incremental syncs?
3. Is there a recommended `timeAfter` configuration or backfill/full-sync approach that can be scheduled automatically for this scenario?
4. If a full/backfill sync is the recommended workaround, how can we configure this in DevLake v1.0.2?
5. Is it possible to configure a backfill/time window that only covers the period containing these long-running workflow runs, rather than performing a complete full sync?
6. Has this behavior been fixed or changed in a newer DevLake version?
7. If a permanent fix is planned, is there a target version in which we could expect this behavior to be addressed?
### Production requirement
This is particularly important for our production environment because our workflows can run for 3–4 hours or longer, while our normal GitHub Workflow Runs sync runs every 2 hours.
For example:
```text
10:00 - Workflow starts
10:00 - DevLake sync runs → workflow is in_progress → DevLake skips it
12:00 - DevLake sync runs again
14:00 - Workflow finally completes
14:00+ - Subsequent incremental sync does not appear to revisit the previously skipped run
### What do you expect to happen
We expect DevLake to retain/track GitHub Actions workflow runs while they are `in_progress` and re-check them during subsequent incremental syncs.
Once the workflow run changes to `completed` in GitHub, DevLake should update the existing workflow run record with its final status and information.
This would ensure that long-running GitHub Actions workflows are not missed by incremental syncs when they are still `in_progress` during the initial collection.
### How to reproduce
1. Configure a GitHub connection in DevLake v1.0.2.
2. Configure the GitHub Workflow Runs sync policy with an incremental sync frequency of every 2 hours (`0 */2 * * *`).
3. Start a GitHub Actions workflow that takes several hours to complete.
4. Run the DevLake "Collect Workflow Runs" task while the GitHub workflow is still `in_progress`.
5. Observe that DevLake logs:
`Skipping run{id: ..., number: ...} with status in_progress`
6. Allow the GitHub Actions workflow to finish and become `completed`.
7. Run the next incremental sync.
8. Observe that the previously skipped workflow run does not appear to be collected or updated in DevLake.
Expected result:
The workflow run should be retained/re-checked and updated once its GitHub status changes from `in_progress` to `completed`.
Actual result:
The workflow run is skipped while `in_progress` and does not appear to be revisited by subsequent incremental syncs after completion.
### Anything else
Skipping run{id: 3372***, number: 1158***} with status in_progress
### Version
v1.0.2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the GitHub “Collect Workflow Runs” task and the workflow collector, focusing on its handling of the GitHub /actions/runs results and incremental syncs. Reproduce the reported sequence with an in_progress run, then a completed run, and verify that the completed workflow is retained or revisited and its final status is available after a later incremental sync.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, github-actions, go
- Domain
- data-engineering, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100