Azure / Azure/azure-sdk-tools

[Tracking] Add code coverage for storage CI check

Open
#15,768 2 comments 0 reactions 0 assignees View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
1d 22h
Merged PRs (30d)
115

Description

## Summary

Add code coverage reporting for storage CI pipelines. This will help track, aggregate, and render coverage results from all supported languages, ensuring actionable feedback and consistency for engineering teams. The table and tasks below describe the current state, required enhancements, and recommended steps per language. This change should also apply to CI pipelines for other services.

### Current Status

| Aspect | Java | .NET | Python | JavaScript | Go | Rust |
|---|---|---|---|---|---|---|
| **Coverage Tool** | JaCoCo | Coverlet | `pytest-cov` + `coverage.py` | Istanbul (via Vitest) | `go test -coverprofile` (built-in) | None |
| **Report Format** | JaCoCo XML + HTML | Cobertura XML | Cobertura XML | Cobertura XML + HTML | Go coverprofile (need conversion to cobertura) | N/A |
| **HTML Report Generation** | ✅ JaCoCo generates HTML | ✅ ReportGenerator | Need fixes (worked before) | ✅ Vitest Istanbul HTML | ❌ None | ❌ None |
| **Aggregation Method** | `generate_aggregate_pom.py` → `jacoco:report-aggregate` | ReportGenerator merges per-project Cobertura files | `create_coverage.py` combines `.coverage_*` files → `coverage combine` | Per-package only (no cross-package merge) | `gocovmerge` to merge per-module coverprofiles | `cargo llvm-cov --workspace` (native workspace aggregation) |
| **Azure DevOps Rendering** | ✅ Working | ✅ Working | Needs fix | ✅ Working | ❌ Not rendering | ❌ Not rendering |

### Estimated Effort and Complexity

| Language | Effort Estimate | Complexity |
|---|---|---|
| Python | 2 day | Low |
| JavaScript | Done | Medium |
| Go | 1 week | Medium (add format conversion + publish) |
| Rust | 2 week | High (build from scratch) |

---

## Task Breakdown
### Python
| Task | Description | Preferred Owner |
|---|---|---|
| Investigate DevOps coverage rendering | Run a CI build and confirm Azure DevOps "Code Coverage" tab populates. | Language team |

### JavaScript / TypeScript
| Task | Description | Preferred Owner | - DONE
|---|---|---|
| Enable coverage in CI test template | Add coverage collection and `PublishCodeCoverageResults` to `eng/pipelines/templates/jobs/ci.tests.yml`. | EngSys team |
| Add CI coverage matrix leg | Update matrix with `PublishCodeCoverage: true` like live test leg. | EngSys team |
| Adjust Vitest coverage condition | Update `vitest.shared.config.ts` to enable coverage for CI variable. | EngSys team |
| Handle browser coverage | Decide if browser Cobertura should be published in Azure DevOps. | |
| Validate rendering | End-to-end validation in CI. | |

### Go
| Task | Description | Preferred Owner |
|---|---|---|
| Merge coverage data | Use `gocovmerge` to merge per-module `coverage.txt` files. | Language team |
| Convert to Cobertura | Use `gocover-cobertura` to convert merged profiles. | Language team |
| Publish to pipeline | Add publish task. | |
| Add matrix leg / gating var | Enable via variable and matrix leg. | |
| Validate rendering | E2E validation in CI. | |

### Rust
| Task | Description | Preferred Owner |
|---|---|---|
| Coverage tool selection | Use `cargo-llvm-cov` for coverage. | Language team |
| Add coverage leg | Use workspace aggregation and Cobertura/HTML generation. | Language team |
| Publish results | Add publish task. | |
| Gate with matrix variable | Add gating and matrix entry. | |
| Validate rendering | E2E validation in CI. | |

---

**Notes:**
- Leverage existing patterns and CI templates (`eng/pipelines/templates`).
- HTML report generation should be enabled where possible for engineering consumption.
- Pipeline overhead should be limited to covered jobs using gating/matrix variables.

Contributor guide

Open the contributing guide

Research direction

Start by reading the task tables and the named CI files under eng/pipelines/templates, especially jobs/ci.tests.yml and vitest.shared.config.ts. Review the existing aggregation references to generate_aggregate_pom.py and create_coverage.py, then run a relevant CI build. Done means supported language coverage is aggregated, published, rendered in Azure DevOps, and gated by the intended matrix variables.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, go, java, javascript, python, rust, typescript
Domain
build-system, ci-cd, devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.