fsprojects / fsprojects/Paket

Recover and re-enable the full Linux integration test suites

Open
#4,346 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

build needs-review
Dominant language
F#
Stars
2.1k
Forks
528
Avg merge
1d 12m
Merged PRs (30d)
54

Description

Context
  • PR https://github.com/fsprojects/Paket/pull/4345 restored GitHub Actions CI and exposed integration-suite rot after prerequisite/restore/build failures were fixed.
  • Diagnostic run https://github.com/fsprojects/Paket/actions/runs/32868698103 ran with .NET SDK 9.0.309, .NET 8 runtime, Mono, EnableWindowsTargeting, and a temporary 120-minute test timeout.
  • net9 result: 13 failed, 203 passed, 10 skipped, 226 total, 10m26s.
  • net461 result: 3 failed, 55 passed, 0 skipped, 58 total, 1h34m.
  • The full suites were removed from PR 4345's required matrix so CI establishment could proceed. The required Linux restore/build/publish/unit tests and Windows remain green. However, the integration suites need proper restoration and re-enablement rather than silent abandonment.

Failure Inventory
net9
  1. #2394 mix dependencies from multiple groups with conditions
  2. fcs can type check
  3. run fsi integration tests that have deterministic output
  4. #2694 paket fixnuspec should not remove project references
  5. #3317 pack multitarget with p2p
  6. #3558 pack multitarget with p2p by tfm
  7. #3983-dont-remove-packed-projects
  8. #4183 writes ranges for floating deps
  9. apply framework restriction for force true/false and NuGet v2/v3 (resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)
  10. apply framework restriction for force true/false and NuGet v2/v3 (resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)
  11. apply framework restriction for force true/false and NuGet v2/v3 (resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)
  12. apply framework restriction for force true/false and NuGet v2/v3 (resolved Microsoft.AspNetCore.WebUtilities 10.0.11 while test expects <9)
  13. #2684 Paket should not be called the second time in msbuild
net461
  • #1783 generates binding redirect when assembly with different version of main group
  • #2394 mix dependencies from multiple groups with conditions
  • #2528 sdk-2: install should not warn with FSharp.Core

Likely Drift Clusters
  • Assembly Metadata / Package Name Output: Package output names now include CI/repository build metadata such as 1.0.0+<commit>.nupkg, while several tests expect exact 1.0.0.nupkg names.
  • Live Feeds & Dependency Resolution: Tests using live NuGet feeds have moved under newer package availability (notably Microsoft.AspNetCore.WebUtilities 10.0.11 is being resolved instead of the expected version).
  • FSI/FCS Changes: FSI/FCS deterministic output and tool-host behavior has evolved.
  • MSBuild / Tooling Behavior: Restore hash / MSBuild behavior changed for #2684.
  • Potential Regressions: #2394 and #4183 may be genuine Paket behavior regressions and should not be fixed by merely modifying expectations.

Were these once passing? (Provenance)

These are regression tests introduced for real behavior and were presumably useful and passing when added. We could not establish an authoritative last-green date because historical Travis/AppVeyor status records are no longer available through GitHub. For historical tracking, here is the introduction evidence:

  • #2394: 62d765fa7 (2024-03-07)
  • FSI/FCS tests and golden suite: 579e4c582 (2020-10-05)
  • #2694: 9496bd4c2 (2019-11-04), fixture history reaches ae4ab31b3 (2017-09-02)
  • #3317 / #3558: 8fb52a72f (2019-06-09)
  • #3983: a0656c310 (2021-02-08)
  • #4183: 3f71e6ce3 (2022-11-09)
  • Framework restriction parameterization: 7fcb5a672 (2025-01-09)
  • #2684: fabcf81aa (2019-07-02)

Asset Recoverability
  • Assets appear recoverable: 99 fixture files across the affected scenarios and 8 FSI .golden files are still tracked in git, with historical versions available from the commits referenced above.
  • Generated integrationtests/scenarios/**/temp, nupkgs, obj/cache files, and FSI .output files are transient or ignored and were not uploaded by the failing run. They can be regenerated locally or via actions, but should be captured as artifacts before changing any assertions.
  • Existing .golden files should only be refreshed after carefully reviewing semantic differences. Do not bulk-accept output.

Proposed Recovery Plan Checklist
  • 1. Reproduce each failure cluster independently using NUnit filters, initially matching SDK 9.0.309/.NET 8/Mono from the diagnostic run.
  • 2. Add a temporary workflow_dispatch/nightly recovery workflow or matrix that uploads TRX, generated .output, relevant scenario temp directories/nupkgs, and binlogs on failure.
  • 3. Compare regenerated assets against current git and git show at each introduction commit; reconstruct an older known-compatible SDK/feed setup where useful to establish the old expected behavior.
  • 4. Classify each failure as product regression, stale expectation, unpinned live-feed dependency, or unsupported legacy net461 behavior.
  • 5. Pin package versions/feed inputs where the test is intended to validate Paket rather than latest NuGet contents; make package-file assertions SemVer-aware when build metadata is irrelevant.
  • 6. Fix product regressions (#2394 / #4183 / #2684 candidates) before updating fixtures.
  • 7. Regenerate goldens only for reviewed intentional FSI/FCS output changes.
  • 8. Shard restored suites so each CI job stays comfortably below 60 minutes, especially net461.
  • 9. Re-enable the recovered suites as required Actions checks (or explicitly document/migrate unsupported net461 coverage).

Acceptance Criteria
  • Every currently failing test is fixed, intentionally updated with rationale, or explicitly retired/migrated with equivalent coverage.
  • net9 integration suite is repeatably green on Actions.
  • net461 is repeatably green or unsupported coverage is migrated/documented.
  • Failure artifacts are retained by CI.
  • Full suites are re-enabled in the Actions matrix without making CI impractically slow/unreliable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the NUnit-filtered integration suites and the diagnostic GitHub Actions matrix using SDK 9.0.309, .NET 8, and Mono. Inspect integrationtests/scenarios/**/temp, tracked fixture files, and the eight FSI .golden files; retain generated outputs and binlogs before changing assertions. Done means each failure is fixed, intentionally updated with rationale, or retired/migrated, and net9/net461 coverage is green and required in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
fsharp, github-actions
Domain
build-system, ci-cd, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.