Fix #10047 SlowBVT failure in distributed directory stack
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 13h 56m
- Merged PRs (30d)
- 351
Description
## Summary
PR #10047 is failing `Test (SlowBVT, windows-latest, net8.0)` in the distributed grain directory stack.
Failing job: https://github.com/dotnet/orleans/actions/runs/25139882594/job/73784372497?pr=10047
Failing test:
```text
Tester.HeterogeneousSilosTests.UpgradeTests.MinimumVersionTests.AlwaysCreateActivationWithMinimumVersion
```
## Observed failure
The CI failure occurs after the test stops the V2 standalone silo and then expects all subsequent activations to resolve to V1. The distributed grain directory lookup still targets the stopped V2 silo and fails with `ConnectionRefused`:
```text
Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S127.0.0.1:24277:136562748
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 127.0.0.1:24277. Error: ConnectionRefused
at Orleans.Runtime.GrainDirectory.DistributedGrainDirectory.Lookup(...)
at Orleans.Runtime.GrainDirectory.CachedGrainLocator.Lookup(...)
at Orleans.Runtime.Placement.PlacementService...
at Tester.HeterogeneousSilosTests.UpgradeTests.UpgradeTestsBase.Step1_StartV1Silo_Step2_StartV2Silo_Step3_StopV2Silo(Int32 step2Version)
```
A local run on the current #10047 head also reproduced a related pre-stop symptom: lookup targeted a V2 partition system target before that target was active:
```text
Orleans.Runtime.OrleansMessageRejectionException: SystemTarget sys.svc.dir.grain.part/... not active on this silo.
```
## Investigation notes
#10047 currently targets `main` and contains only:
1. `Fix directory snapshot transfer ranges`
2. `Stabilize directory ownership checks`
That means its CI runs without the prerequisite distributed-directory upgrade-resilience commits which are present elsewhere in the stack, including:
- `DistributedGrainDirectory : IRemoteGrainDirectory`
- `Add distributed grain directory rolling upgrade tests`
- `Improve DistributedRemoteGrainDirectory rolling upgrade resilience`
- `Fix mixed local-distributed rolling upgrade`
- `Batch large directory transfer payloads`
The same focused versioning test passed locally on the top-of-stack branch which includes those prerequisite commits.
I also tried a standalone test-hardening branch based on current `main` which configured the standalone versioning silos to use faster one-vote liveness (`NumMissedProbesLimit = 1`, `NumVotesForDeathDeclaration = 1`, `ProbeTimeout = 1s`) and made the test wait using the same membership options. That is not sufficient as an independent `main` fix: current `main` still fails the focused test with `SystemTarget ... not active on this silo`.
## Proposed resolution
Restore #10047 to the intended stack shape instead of running it directly against `main`:
1. Rebase/recreate `fix/directory-snapshot-transfer-ranges` on top of its prerequisite stack branch, currently `fix/directory-transfer-payload-batching` / PR #10053.
2. Retarget #10047 back to that prerequisite branch while it is under review.
3. Keep only the snapshot-transfer-range commit in #10047 if possible; keep `Stabilize directory ownership checks` in the next layer.
4. Consider keeping the standalone versioning-test liveness hardening in the lower distributed-directory compatibility layer, since these tests run external two-silo clusters and otherwise wait using default membership settings which are too slow/brittle for this failure mode.
Until the prerequisite commits are included in the PR merge base, #10047's SlowBVT result is not representative of the intended directory migration stack.
Contributor guide
Research direction
Start with Tester.HeterogeneousSilosTests.UpgradeTests.MinimumVersionTests.AlwaysCreateActivation and the SlowBVT job for PR #10047. Review the prerequisite distributed-directory changes in PR #10053 and the referenced rolling-upgrade commits before examining the branch merge base. Done means #10047 is layered on the prerequisite branch, contains the intended snapshot-transfer change, and the focused test no longer fails against the incomplete stack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, github-actions
- Domain
- ci-cd, distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100