dotnet / dotnet/machinelearning

Fix and re-enable skipped and flaky tests

Open
#7,698 1 comment 0 reactions 0 assignees View on GitHub
area-Infrastructure untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Restore test coverage lost to temporary skips, known failures, missing test assets, and unresolved platform differences.

This issue tracks the work to investigate the remaining exclusions, fix their underlying causes, and re-enable the affected tests. Existing issues should remain the home for detailed investigation and fixes, with progress tracked here.

## Skipped tests

- [ ] #5845: Resolve the remaining Helix-related exclusions, including `TestDownloadError`, `MatrixFactorizationSimpleTrainAndPredict`, quantile regression, and factorisation-machine initialisation tests. Reconcile the older checklist with current source.
- [ ] #6684: Re-enable `MulticlassLRTest` on affected architectures and `TestGridSearchTrialRunner2`. The latter currently has an unconditional skip despite its architecture-specific explanation. Investigate the related failure reports in #6841, #6842, and #6843 without assuming they share a root cause.
- [ ] #6043: Restore the image conversion tests excluded on ARM for stability.
- [ ] #7688: Add the missing Windows ARM baseline and remove the corresponding early return in `BinaryClassifierSymSgdTest`. Investigate its separate Linux early return as well.
- [ ] #6978: Address the memory requirements of `QATests.TestSimpleQA` and restore automated coverage.
- [ ] #1120: Provide suitable time-series data for `EntryPointSsaChangePoint` and `EntryPointSsaSpikeDetector`; address the equivalent dataset-related skip in `SavePipeSsaSpike`.
- [ ] #1391: Resolve or explicitly retire the outstanding x86 exclusion for `BenchmarksProjectIsNotBroken`, according to the supported benchmark configurations.

## Flaky tests

- [ ] #7419: `TestAppendMany_SizeMoreThanMaxBufferCapacity` triggers the OOM killer.
- [ ] #7418: AutoML experiment cancellation test fails in CI.
- [ ] #7393: Concurrent `DownloadImageSet` calls contend for shared files.
- [ ] #7067: AutoML writes test output after the test has finished.
- [ ] #7060: Assertion failure in `TestEstimatorMulticlassNaiveBayesTrainer`.
- [ ] #7057: Incorrect results in `NaiveCalibratorOnnxConversionTest`.
- [ ] #6996: Possible deadlock in `Microsoft.ML.Tests`.
- [ ] #6971: SQL connection timeouts in `DatabaseLoaderTests`.
- [ ] #6961: LightGBM allocation failure and process crash.
- [ ] #6960: Access violation during `TestRegressionScenario`.
- [ ] #2299: TBD
- [ ] #772

Contributor guide

Open the contributing guide

Research direction

Start with one existing issue linked from this tracker, such as #6043 or #7688, and inspect the named tests and their skip or early-return conditions. Run the affected test on the relevant platform or CI configuration, then trace the linked failure reports. Done means the underlying failure, missing asset, or platform condition is addressed and the test is re-enabled with coverage restored.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.