Recover and re-enable the full Linux integration test suites
Nobody has claimed this yet.
- Dominant language
- F#
- Stars
- 2.1k
- Forks
- 528
- Avg merge
- 1d 12m
- Merged PRs (30d)
- 54
Description
Context
- PR https://github.com/fsprojects/Paket/pull/4345 restored GitHub Actions CI and exposed integration-suite rot after prerequisite/restore/build failures were fixed.
- Diagnostic run https://github.com/fsprojects/Paket/actions/runs/32868698103 ran with .NET SDK 9.0.309, .NET 8 runtime, Mono, EnableWindowsTargeting, and a temporary 120-minute test timeout.
- net9 result: 13 failed, 203 passed, 10 skipped, 226 total, 10m26s.
- net461 result: 3 failed, 55 passed, 0 skipped, 58 total, 1h34m.
- The full suites were removed from PR 4345's required matrix so CI establishment could proceed. The required Linux restore/build/publish/unit tests and Windows remain green. However, the integration suites need proper restoration and re-enablement rather than silent abandonment.
Failure Inventory
net9
#2394 mix dependencies from multiple groups with conditionsfcs can type checkrun fsi integration tests that have deterministic output#2694 paket fixnuspec should not remove project references#3317 pack multitarget with p2p#3558 pack multitarget with p2p by tfm#3983-dont-remove-packed-projects#4183 writes ranges for floating depsapply framework restriction for force true/false and NuGet v2/v3(resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)apply framework restriction for force true/false and NuGet v2/v3(resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)apply framework restriction for force true/false and NuGet v2/v3(resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)apply framework restriction for force true/false and NuGet v2/v3(resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)#2684 Paket should not be called the second time in msbuild
net461
#1783 generates binding redirect when assembly with different version of main group#2394 mix dependencies from multiple groups with conditions#2528 sdk-2: install should not warn with FSharp.Core
Likely Drift Clusters
- Assembly Metadata / Package Name Output: Package output names now include CI/repository build metadata such as
1.0.0+<commit>.nupkg, while several tests expect exact1.0.0.nupkgnames. - Live Feeds & Dependency Resolution: Tests using live NuGet feeds have moved under newer package availability (notably Microsoft.AspNetCore.WebUtilities 10.0.11 is being resolved instead of the expected version).
- FSI/FCS Changes: FSI/FCS deterministic output and tool-host behavior has evolved.
- MSBuild / Tooling Behavior: Restore hash / MSBuild behavior changed for
#2684. - Potential Regressions:
#2394and#4183may be genuine Paket behavior regressions and should not be fixed by merely modifying expectations.
Were these once passing? (Provenance)
These are regression tests introduced for real behavior and were presumably useful and passing when added. We could not establish an authoritative last-green date because historical Travis/AppVeyor status records are no longer available through GitHub. For historical tracking, here is the introduction evidence:
- #2394:
62d765fa7(2024-03-07) - FSI/FCS tests and golden suite:
579e4c582(2020-10-05) - #2694:
9496bd4c2(2019-11-04), fixture history reachesae4ab31b3(2017-09-02) - #3317 / #3558:
8fb52a72f(2019-06-09) - #3983:
a0656c310(2021-02-08) - #4183:
3f71e6ce3(2022-11-09) - Framework restriction parameterization:
7fcb5a672(2025-01-09) - #2684:
fabcf81aa(2019-07-02)
Asset Recoverability
- Assets appear recoverable: 99 fixture files across the affected scenarios and 8 FSI
.goldenfiles are still tracked in git, with historical versions available from the commits referenced above. - Generated
integrationtests/scenarios/**/temp, nupkgs, obj/cache files, and FSI.outputfiles are transient or ignored and were not uploaded by the failing run. They can be regenerated locally or via actions, but should be captured as artifacts before changing any assertions. - Existing
.goldenfiles should only be refreshed after carefully reviewing semantic differences. Do not bulk-accept output.
Proposed Recovery Plan Checklist
- 1. Reproduce each failure cluster independently using NUnit filters, initially matching SDK 9.0.309/.NET 8/Mono from the diagnostic run.
- 2. Add a temporary workflow_dispatch/nightly recovery workflow or matrix that uploads TRX, generated
.output, relevant scenariotempdirectories/nupkgs, and binlogs on failure. - 3. Compare regenerated assets against current git and
git showat each introduction commit; reconstruct an older known-compatible SDK/feed setup where useful to establish the old expected behavior. - 4. Classify each failure as product regression, stale expectation, unpinned live-feed dependency, or unsupported legacy net461 behavior.
- 5. Pin package versions/feed inputs where the test is intended to validate Paket rather than latest NuGet contents; make package-file assertions SemVer-aware when build metadata is irrelevant.
- 6. Fix product regressions (
#2394/#4183/#2684candidates) before updating fixtures. - 7. Regenerate goldens only for reviewed intentional FSI/FCS output changes.
- 8. Shard restored suites so each CI job stays comfortably below 60 minutes, especially net461.
- 9. Re-enable the recovered suites as required Actions checks (or explicitly document/migrate unsupported net461 coverage).
Acceptance Criteria
- Every currently failing test is fixed, intentionally updated with rationale, or explicitly retired/migrated with equivalent coverage.
net9integration suite is repeatably green on Actions.net461is repeatably green or unsupported coverage is migrated/documented.- Failure artifacts are retained by CI.
- Full suites are re-enabled in the Actions matrix without making CI impractically slow/unreliable.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the NUnit-filtered integration suites and the diagnostic GitHub Actions matrix using SDK 9.0.309, .NET 8, and Mono. Inspect integrationtests/scenarios/**/temp, tracked fixture files, and the eight FSI .golden files; retain generated outputs and binlogs before changing assertions. Done means each failure is fixed, intentionally updated with rationale, or retired/migrated, and net9/net461 coverage is green and required in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fsharp, github-actions
- Domain
- build-system, ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100