Lightning-AI / Lightning-AI/utilities

Collect and unify pytest results across different runs

Open
#6 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ci/cd enhancement help wanted
Dominant language
Python
Stars
62
Forks
23
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature
Create a workflow that collects and merges pytest results from CI runs across different operating systems, acclerators and software versions.

With this feature implemented, we will be able to see a merged list of all test cases succeeded/failed/skipped across all CI configurations.

### Motivation
We have tests running on across different OS, accelerators and software versions, and currently, each CI run has its own result only, which making it almost impossible to monitor which tests are running or skipped across all such configurations.

Due to this, we've experienced an issue a while ago due to the lack of observability where all of the horovod tests had never run for a long time of period in PL repo.

### Pitch
To be explored.
(I guess we could somehow utilise https://github.com/pytest-dev/pytest-reportlog)

### Alternatives
To be explored.

### Additional context
Codecov automatically merges coverage results uploaded from different CI runs: https://app.codecov.io/gh/Lightning-AI/lightning/
AFAIK, cov result doesn't hold any pytest results, so need to find another way to collect each test case status from different CI settings.

Open for any suggestions 💜

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the repository's existing GitHub Actions CI configurations and how pytest results are produced across operating systems, accelerators, and software versions. Compare available collection approaches, including pytest-reportlog, and define done as a merged view showing each test case as succeeded, failed, or skipped across all CI configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.