[Task] Harden periodic tasks so a single exception no longer cancels the schedule
- Dominant language
- Java
- Stars
- 8.8k
- Forks
- 3.1k
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 85
Description
## Background
A 2026-08-02 source audit confirmed that 8 open issues share the **same root cause**: a periodic task scheduled via `ScheduledThreadPoolExecutor` (either `scheduleAtFixedRate` or `scheduleWithFixedDelay`) is permanently cancelled after a single uncaught exception, because the JDK contract states *"if any execution of the task encounters an exception, subsequent executions are suppressed."* Confirmed that `org.apache.shenyu.common.exception.ShenyuException extends RuntimeException`, so rethrows escape the task and kill the schedule.
This one defect class silently stops cluster election, instance health sync, registry watches, token refresh, heartbeat, and the upstream health-check loop — a broad class of "the gateway/admin silently stops reconciling" symptoms.
## Affected issues (all verified VALID)
- #6487 Client heartbeat scheduler can stop permanently after one full failure
- #6495 Cluster master election scheduler can stop after one renewal exception
- #6496 Instance health DB sync scheduler can stop after one persistence error
- #6498 Consul TTL heartbeat task can stop after one transient Consul error
- #6499 Consul instance watch task can stop after one polling error
- #6500 Eureka instance watch task can stop after one polling error
- #6502 Upstream health check can get stuck after one async check failure
- #6504 HTTP sync token refresh can stop after a login response parsing error
## Variants observed
- **No try/catch at all** (task body throws directly): #6487, #6498, #6499.
- **Catch-and-rethrow as `ShenyuException`/`RuntimeException`** (still escapes and cancels): #6495, #6500.
- **Sibling task is guarded but this one isn't**: #6496 (`scheduled()` is wrapped, `syncDB()` is not).
- **Aggregation/list bookkeeping fails before `clear()`**: #6502 (`CompletableFuture.allOf(...).join()` re-throws before `futures.clear()`).
## Suggested fix pattern (single mechanism closes all 8)
Introduce a common resilient wrapper for periodic tasks in `shenyu-common` (or a small helper in each module) that:
1. Wraps the task body in `try { ... } catch (Throwable t) { LOG.error(...); }` so an exception never escapes and cancels the schedule.
2. For tasks that must surface fatal errors, swallow-and-log for transient errors and escalate (metric/alert) rather than killing the schedule.
3. Ensures any shared mutable state (e.g. the `futures` list in #6502) is cleared in a `finally`, independent of whether `join()` succeeded.
Apply it uniformly to `ShenyuClusterService`, `InstanceCheckService`, `TtlScheduler`, `ConsulInstanceRegisterRepository`, `EurekaInstanceRegisterRepository`, `ShenyuClientURIExecutorSubscriber`, `AccessTokenManager`, and `UpstreamCheckService`.
## Goal
Track the shared fix so the 8 issues can be closed by one well-scoped PR (or a small series) rather than eight independent patches that risk re-implementing the same guard inconsistently.
_Audit ref: docs/issue-candidates-2026-08-02.md_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with docs/issue-candidates-2026-08-02.md and inspect the scheduled methods in ShenyuClusterService, InstanceCheckService, TtlScheduler, the registry repositories, ShenyuClientURIExecutorSubscriber, AccessTokenManager, and UpstreamCheckService. Trace where exceptions escape and where shared state is updated. Done means one consistent guard covers the eight affected tasks, exceptions no longer cancel schedules, and the futures bookkeeping is cleared even after failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100