dotnet / dotnet/orleans

Fix #10047 SlowBVT failure in distributed directory stack

Open
#10,075 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

## Summary

PR #10047 is failing `Test (SlowBVT, windows-latest, net8.0)` in the distributed grain directory stack.

Failing job: https://github.com/dotnet/orleans/actions/runs/25139882594/job/73784372497?pr=10047

Failing test:

```text
Tester.HeterogeneousSilosTests.UpgradeTests.MinimumVersionTests.AlwaysCreateActivationWithMinimumVersion
```

## Observed failure

The CI failure occurs after the test stops the V2 standalone silo and then expects all subsequent activations to resolve to V1. The distributed grain directory lookup still targets the stopped V2 silo and fails with `ConnectionRefused`:

```text
Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S127.0.0.1:24277:136562748
---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 127.0.0.1:24277. Error: ConnectionRefused
at Orleans.Runtime.GrainDirectory.DistributedGrainDirectory.Lookup(...)
at Orleans.Runtime.GrainDirectory.CachedGrainLocator.Lookup(...)
at Orleans.Runtime.Placement.PlacementService...
at Tester.HeterogeneousSilosTests.UpgradeTests.UpgradeTestsBase.Step1_StartV1Silo_Step2_StartV2Silo_Step3_StopV2Silo(Int32 step2Version)
```

A local run on the current #10047 head also reproduced a related pre-stop symptom: lookup targeted a V2 partition system target before that target was active:

```text
Orleans.Runtime.OrleansMessageRejectionException: SystemTarget sys.svc.dir.grain.part/... not active on this silo.
```

## Investigation notes

#10047 currently targets `main` and contains only:

1. `Fix directory snapshot transfer ranges`
2. `Stabilize directory ownership checks`

That means its CI runs without the prerequisite distributed-directory upgrade-resilience commits which are present elsewhere in the stack, including:

- `DistributedGrainDirectory : IRemoteGrainDirectory`
- `Add distributed grain directory rolling upgrade tests`
- `Improve DistributedRemoteGrainDirectory rolling upgrade resilience`
- `Fix mixed local-distributed rolling upgrade`
- `Batch large directory transfer payloads`

The same focused versioning test passed locally on the top-of-stack branch which includes those prerequisite commits.

I also tried a standalone test-hardening branch based on current `main` which configured the standalone versioning silos to use faster one-vote liveness (`NumMissedProbesLimit = 1`, `NumVotesForDeathDeclaration = 1`, `ProbeTimeout = 1s`) and made the test wait using the same membership options. That is not sufficient as an independent `main` fix: current `main` still fails the focused test with `SystemTarget ... not active on this silo`.

## Proposed resolution

Restore #10047 to the intended stack shape instead of running it directly against `main`:

1. Rebase/recreate `fix/directory-snapshot-transfer-ranges` on top of its prerequisite stack branch, currently `fix/directory-transfer-payload-batching` / PR #10053.
2. Retarget #10047 back to that prerequisite branch while it is under review.
3. Keep only the snapshot-transfer-range commit in #10047 if possible; keep `Stabilize directory ownership checks` in the next layer.
4. Consider keeping the standalone versioning-test liveness hardening in the lower distributed-directory compatibility layer, since these tests run external two-silo clusters and otherwise wait using default membership settings which are too slow/brittle for this failure mode.

Until the prerequisite commits are included in the PR merge base, #10047's SlowBVT result is not representative of the intended directory migration stack.

Contributor guide

Open the contributing guide

Research direction

Start with Tester.HeterogeneousSilosTests.UpgradeTests.MinimumVersionTests.AlwaysCreateActivation and the SlowBVT job for PR #10047. Review the prerequisite distributed-directory changes in PR #10053 and the referenced rolling-upgrade commits before examining the branch merge base. Done means #10047 is layered on the prerequisite branch, contains the intended snapshot-transfer change, and the focused test no longer fails against the incomplete stack.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, github-actions
Domain
ci-cd, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.