microsoft / microsoft/aspire.dev
Retry failed YouTube WebSub subscriptions promptly after a successful redeploy
Nobody has claimed this yet.
- Dominant language
- MDX
- Stars
- 193
- Forks
- 87
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 73
Description
## Problem
A successful site redeploy can inherit a long YouTube WebSub retry delay from the previous deployment. The failure count and `RetryAt` are persisted in Redis, and the first tick of a newly started leader still honors that deadline. After deploying a potential fix, we can therefore wait for the remainder of the capped, approximately one-hour backoff before exercising the changed code.
This is a **retry-eligibility problem**, not a reason to reset the distributed leadership lease or discard a valid Google subscription lease. Persisted backoff protects against retry storms and should remain the normal behavior.
Follow-up to #1685; preserve the distributed backoff protections introduced in #1674.
## Production evidence
Read-only observations on September 17, 2026, between 19:50:30 and 19:50:56 UTC:
- The YouTube leadership key's TTL was extended twice during the observation, so a missing leadership lease did not explain the delay at that time.
- Subscription state had **43 accumulated failures**, compared with an earlier log reporting attempt 39. Attempts had continued; this was not evidence that the worker had stopped.
- There was no pending subscription request, active topic, or recent verification confirmation.
- The persisted next retry deadline was **20:06:54.739 UTC**, approximately 16 minutes after inspection.
Earlier real hub requests returned HTTP 503. The latest failure's HTTP status was not available in the retrieved logs. A deployment-aware retry would improve recovery and diagnosis; it is not an established fix for Google's 503 responses.
## Proposed behavior
Allow **one coordinated, expedited subscription attempt after a new successful deployment**, when a failed subscription or overdue renewal would otherwise be held behind an older retry deadline.
- Run the attempt through the existing leader and subscription-reservation mechanisms, on the first eligible worker tick after the deployment is ready. Do not introduce an independent subscription sender.
- Claim the expedited attempt atomically in shared state. Use a reliable deployment/recovery identifier, not a random process or leader identifier. Concurrent replicas and overlapping old/new revisions must not repeatedly unlock attempts.
- Preserve valid subscription leases and pending verification requests. Do not flush Redis, delete leadership keys, weaken callback validation, or treat HTTP acceptance as successful verification.
- Keep failure history. If the expedited attempt fails, resume normal persisted backoff rather than resetting the counter and starting a fresh rapid retry sequence.
- Define how planned configuration restarts differ from crash loops, scale-out, and ordinary leadership changes. Repeated restarts must not become a way to generate unbounded attempts.
- Account for explicit provider retry constraints when deciding whether an expedited attempt is eligible. Do not increase normal YouTube Data API discovery polling or its request budget.
## Acceptance criteria
- [ ] A new ready deployment can make one eligible recovery attempt without waiting out the previous deployment's full retry delay.
- [ ] The attempt budget is shared across replicas and remains bounded during rolling deployment, leader handoff, and repeated process restarts.
- [ ] Healthy subscriptions, valid pending callbacks, leadership ownership, and unrelated Redis state are preserved.
- [ ] Failure returns to normal persisted backoff; only a valid verification callback establishes or renews the subscription.
- [ ] Safe diagnostics explain whether startup recovery was allowed or deferred, including the reason and next eligible time, without logging tokens or raw subscription state.
- [ ] Deterministic tests cover future retry deadlines, concurrent replicas, overlapping revisions, crash loops, leadership loss, pending verification across restart, healthy active leases, and failed expedited attempts.
- [ ] Operational documentation explains deployment/restart behavior and expected recovery timing.
## Relevant code
- `src/statichost/StaticHost/Live/YouTube/YouTubeWebSubService.cs`: `RunLeaderAsync`, `TickAsync`.
- `src/statichost/StaticHost/Live/YouTube/YouTubeWebSubSubscriptionState.cs`: `YouTubeWebSubSubscriptionTransitions.TryBegin`, `MarkRequestFailed`.
- `src/statichost/StaticHost/Live/YouTube/RedisYouTubeWebSubSubscriptionState.cs`: persisted state and optimistic concurrency.
- `src/statichost/StaticHost/Live/LiveStatusCoordination.cs`: distributed leadership.
- `src/statichost/StaticHost/Live/README.md`: operational guidance.
## Non-goals
No production state reset as part of filing this issue. Do not change the hub/topic URLs, remove verification safeguards, or claim that redeployment itself repairs the external hub failure.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RunLeaderAsync and TickAsync in src/statichost/StaticHost/Live/YouTube/YouTubeWebSubService.cs, then read the transition and persisted-state methods in YouTubeWebSubSubscriptionState.cs and RedisYouTubeWebSubSubscriptionState.cs. Review LiveStatusCoordination.cs and the existing guidance in src/statichost/StaticHost/Live/README.md. Done means bounded coordinated recovery, preserved leases and backoff, deterministic coverage of the listed restart scenarios, safe diagnostics, and updated operational documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- backend, distributed-systems, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100