dotnet / dotnet/machinelearning
Fix and re-enable skipped and flaky tests
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Restore test coverage lost to temporary skips, known failures, missing test assets, and unresolved platform differences.
This issue tracks the work to investigate the remaining exclusions, fix their underlying causes, and re-enable the affected tests. Existing issues should remain the home for detailed investigation and fixes, with progress tracked here.
## Skipped tests
- [ ] #5845: Resolve the remaining Helix-related exclusions, including `TestDownloadError`, `MatrixFactorizationSimpleTrainAndPredict`, quantile regression, and factorisation-machine initialisation tests. Reconcile the older checklist with current source.
- [ ] #6684: Re-enable `MulticlassLRTest` on affected architectures and `TestGridSearchTrialRunner2`. The latter currently has an unconditional skip despite its architecture-specific explanation. Investigate the related failure reports in #6841, #6842, and #6843 without assuming they share a root cause.
- [ ] #6043: Restore the image conversion tests excluded on ARM for stability.
- [ ] #7688: Add the missing Windows ARM baseline and remove the corresponding early return in `BinaryClassifierSymSgdTest`. Investigate its separate Linux early return as well.
- [ ] #6978: Address the memory requirements of `QATests.TestSimpleQA` and restore automated coverage.
- [ ] #1120: Provide suitable time-series data for `EntryPointSsaChangePoint` and `EntryPointSsaSpikeDetector`; address the equivalent dataset-related skip in `SavePipeSsaSpike`.
- [ ] #1391: Resolve or explicitly retire the outstanding x86 exclusion for `BenchmarksProjectIsNotBroken`, according to the supported benchmark configurations.
## Flaky tests
- [ ] #7419: `TestAppendMany_SizeMoreThanMaxBufferCapacity` triggers the OOM killer.
- [ ] #7418: AutoML experiment cancellation test fails in CI.
- [ ] #7393: Concurrent `DownloadImageSet` calls contend for shared files.
- [ ] #7067: AutoML writes test output after the test has finished.
- [ ] #7060: Assertion failure in `TestEstimatorMulticlassNaiveBayesTrainer`.
- [ ] #7057: Incorrect results in `NaiveCalibratorOnnxConversionTest`.
- [ ] #6996: Possible deadlock in `Microsoft.ML.Tests`.
- [ ] #6971: SQL connection timeouts in `DatabaseLoaderTests`.
- [ ] #6961: LightGBM allocation failure and process crash.
- [ ] #6960: Access violation during `TestRegressionScenario`.
- [ ] #2299: TBD
- [ ] #772
Contributor guide
Research direction
Start with one existing issue linked from this tracker, such as #6043 or #7688, and inspect the named tests and their skip or early-return conditions. Run the affected test on the relevant platform or CI configuration, then trace the linked failure reports. Done means the underlying failure, missing asset, or platform condition is addressed and the test is re-enabled with coverage restored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100