Lightning-AI / Lightning-AI/utilities
Collect and unify pytest results across different runs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62
- Forks
- 23
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Create a workflow that collects and merges pytest results from CI runs across different operating systems, acclerators and software versions.
With this feature implemented, we will be able to see a merged list of all test cases succeeded/failed/skipped across all CI configurations.
### Motivation
We have tests running on across different OS, accelerators and software versions, and currently, each CI run has its own result only, which making it almost impossible to monitor which tests are running or skipped across all such configurations.
Due to this, we've experienced an issue a while ago due to the lack of observability where all of the horovod tests had never run for a long time of period in PL repo.
### Pitch
To be explored.
(I guess we could somehow utilise https://github.com/pytest-dev/pytest-reportlog)
### Alternatives
To be explored.
### Additional context
Codecov automatically merges coverage results uploaded from different CI runs: https://app.codecov.io/gh/Lightning-AI/lightning/
AFAIK, cov result doesn't hold any pytest results, so need to find another way to collect each test case status from different CI settings.
Open for any suggestions 💜
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the repository's existing GitHub Actions CI configurations and how pytest results are produced across operating systems, accelerators, and software versions. Compare available collection approaches, including pytest-reportlog, and define done as a merged view showing each test case as succeeded, failed, or skipped across all CI configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100