dotnet / dotnet/fsharp

Flaky test: FCS tooltip `#r` reference-assembly resolution is non-deterministic (Fsx.Bug5073, ResolveFromGAC)

Open
#20,276 0 comments 0 reactions 0 assignees View on GitHub
Bug Needs-Triage
Dominant language
F#
Stars
4.3k
Forks
876
Avg merge
4d 11h
Merged PRs (30d)
131

Description

**Description**

A couple of `#r` reference-assembly *tooltip* tests in `tests/FSharp.Compiler.Service.Tests/Tooltip/TooltipTests.Declarations.fs` intermittently produce an **empty tooltip** in the FCS test host, so the `assertReferenceTooltipContains` assertions flake:

- `Fsx.Bug5073` — `#r "System"`, expects the tooltip to contain `Reference Assemblies\Microsoft`
- `Fsx.HashR_QuickInfo.ResolveFromGAC` — `#r "CustomMarshalers"`, expects `.NETFramework` / `CustomMarshalers.dll`

The root cause looks like non-deterministic reference-assembly / GAC name resolution inside the FCS test host rather than a real product regression (the same `#r` resolves fine in VS/`dotnet fsi`).

**Current state**

- `ResolveFromGAC` is already skipped on `main` with `[]`.
- `Fsx.Bug5073` was briefly skipped in PR #20100 (the maestro Roslyn dependency bump) to keep that PR green, but the skip was reverted so the flake isn't hidden inside an unrelated dependency update. This issue tracks it instead.

**Repro steps**

1. Run `dotnet test tests/FSharp.Compiler.Service.Tests --filter "FullyQualifiedName~Fsx.Bug5073"` (Windows/Desktop; these are `FactForDESKTOP`).
2. Re-run several times — the tooltip is occasionally empty and the assertion fails.

**Expected behavior**

The reference tooltip for `#r "System"` / `#r "CustomMarshalers"` resolves deterministically and the tests pass reliably.

**Actual behavior**

Intermittently empty tooltip → `assertReferenceTooltipContains` fails.

**Suggested fix**

Make the test host's reference-assembly/GAC resolution deterministic (or harden the tooltip resolution / add a retry-until-ready in the test helper), then re-enable both tests.

**Known workarounds**

Skip the tests (undesirable — hides the flake).

Contributor guide

Open the contributing guide

Research direction

Start in tests/FSharp.Compiler.Service.Tests/Tooltip/TooltipTests.Declarations.fs and run the documented dotnet test filter on Windows/Desktop, repeating it to reproduce the empty tooltip. Compare Fsx.Bug5073 and the skipped Fsx.HashR_QuickInfo.ResolveFromGAC while tracing reference-assembly/GAC resolution in the FCS test host. Done means both tests resolve their expected tooltip text deterministically and can be re-enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
fsharp
Domain
compilers, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.