[tests] Random BundleStructureWithRemoteMac failure from timestamp churn in shared Windows SDK
- Dominant language
- C#
- Stars
- 2.9k
- Forks
- 576
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 123
Description
### Summary
`Xamarin.Tests.WindowsTest.BundleStructureWithRemoteMac(iOS, "ios-arm64", All, "Debug")` is randomly failing in the Windows remote test leg.
The failure shows up as either:
- `Rebuild 1: shared-dotnet.plist: timestamp changed ...`
- or `_CompileAppManifest` unexpectedly executing on what should be a no-op rebuild after only touching `AppDelegate.cs`.
Example recent failure:
- DevDiv build: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=13839857&view=results
### Diagnostic work
I investigated this using:
- the additional logging added in `0b8b50e94f5226156d582162024901dabd6c6b11`
- the failing HTML report / vsdrop artifacts from build `13839857`
- the relevant binlogs (`windows-remote-dotnet-tests.binlog`, the `BundleStructure` iOS build binlogs, and `install-workloads.binlog`)
The chain from the failing artifacts is:
1. `obj/.../unpack/bindings-framework-test/PartialAppManifest/shared-dotnet.plist` gets a newer timestamp.
2. That makes `_CompileAppManifest` rerun, even though the test only touched `AppDelegate.cs`.
3. The plist is rewritten because `_BeforeUnpackLibraryResources` deletes the resource stamp files, so `_UnpackLibraryResources` runs again.
4. The stamp files are deleted because `bindings-framework-test.dll` is newer than its stamp.
5. `bindings-framework-test.dll` is rebuilt because `_GenerateBindings`/`BGen` reruns and rewrites `FrameworkTest.g.cs`.
6. `BGen` reruns because one of its references (`MonoTouch.Dialog.dll`) rebuilt.
7. `MonoTouch.Dialog.dll` rebuilt because MSBuild considered `tests/dotnet/Windows/bin/dotnet/sdk/.../Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll` newer than the previous output.
Notably, this analyzer-driven invalidation already happens in the **initial** build, before the test performs the rebuild step, so it doesn't look like `Rebuild 1` itself is what changed that analyzer timestamp.
### Conclusions
This looks like an incremental-build / timestamp churn issue in the shared repo-local Windows SDK tree (`tests/dotnet/Windows/bin/dotnet`), not a semantic change in the `BundleStructure` project itself.
Two things made the flake visible:
- unchanged unpacked resources were being rewritten, which bumped the timestamp of `shared-dotnet.plist`
- unchanged `bgen` outputs were being rewritten, which made the binding project look changed and caused the stamp invalidation chain to continue
The remaining open question is why the analyzer DLL in the shared SDK tree has a later mtime than the referenced-project outputs at test start.
A likely contributor is that the job mutates that shared SDK tree as part of workload installation. The `install-workloads.binlog` shows:
```text
D:\AzDO\_work\17\s\macios\tests\dotnet\Windows\bin\dotnet\dotnet workload install ios tvos maccatalyst macos ...
```
So all Windows unit tests are building against a mutable, repo-local SDK installation, and that may be enough to leave SDK inputs newer than previously built referenced projects.
### Suggested next steps
1. Confirm what step is refreshing or rewriting files under `tests/dotnet/Windows/bin/dotnet/sdk/...` and whether that can happen between test runs.
2. Consider isolating workload-installing steps/tests from the rest of the Windows unit tests, or stop sharing the same mutable SDK tree.
3. Keep the extra logging in `WindowsTest` so the next failure shows the same file/target chain if this still happens.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.